Re: [dpdk-dev] [PATCH v16 00/23] Add DLB PMD

[Jerin Jacob <jerinjacobk@gmail.com>]

On Mon, Nov 2, 2020 at 4:58 AM Timothy McDaniel
<timothy.mcdaniel@intel.com> wrote:
>
> The following patch series adds support for a new eventdev PMD. The DLB
> PMD adds support for the Intel Dynamic Load Balancer (DLB) hardware.
> The DLB is a PCIe device that provides load-balanced, prioritized
> scheduling of core-to-core communication. The device consists of
> queues and arbiters that connect producer and consumer cores, and
> implements load-balanced queueing features including:
> - Lock-free multi-producer/multi-consumer operation.
> - Multiple priority levels for varying traffic types.
> - 'Direct' traffic (i.e. multi-producer/single-consumer)
> - Simple unordered load-balanced distribution.
> - Atomic lock-free load balancing across multiple consumers.
> - Queue element reordering feature allowing ordered load-balanced
>   distribution.
>
> The DLB hardware supports both load balanced and directed ports and
> queues. Unlike other eventdev devices already in the repo,  not all
> DLB ports and queues are equally capable. In particular, directed
> ports are limited to a single link, and must be connected to a directed
> queue.
> Additionally, even though LDB ports may link multiple queues, the
> number of queues that may be linked is limited by hardware. Another
> difference is that DLB does not have a straightforward way of carrying
> the flow_id in the queue elements (QE) that the hardware operates on.
>
> While reviewing the code, please be aware that this PMD has full
> control over the DLB hardware. Intel will be extending the DLB PMD
> in the future (not as part of this first series) with a mode that we
> refer to as the bifurcated PMD. The bifurcated PMD communicates with a
> kernel driver to configure the device, ports, and queues, and memory
> maps device MMIO so datapath operations occur purely in user-space.
>
> The framework to support both the PF PMD and bifurcated PMD exists in
> this patchset, and is why the iface.[ch] layer is present.


Series applied to dpdk-next-eventdev/for-main with the following fix. Thanks.

diff --git a/doc/guides/eventdevs/dlb.rst b/doc/guides/eventdevs/dlb.rst
index d44afcdcf..4c4f56b2b 100644
--- a/doc/guides/eventdevs/dlb.rst
+++ b/doc/guides/eventdevs/dlb.rst
@@ -2,7 +2,7 @@
     Copyright(c) 2020 Intel Corporation.

 Driver for the Intel® Dynamic Load Balancer (DLB)
+=================================================
-==================================================

 The DPDK dlb poll mode driver supports the Intel® Dynamic Load Balancer.



>
> Major changes in V16
> ====================
> Address additional comments from David Marchand:
> - converted printfs in dlb/pf/dlb_main.c to DLB_LOG
> - fixed a repeated word error in dlb/pf/base/osdep_bitmap.h
> - caught up with marking the patches that Gage reviewed
>
> Major changes in V15
> ====================
> Address comments from David Marchand:
> - this patch-set is based on Nov 1, 2020 dpdk-next-eventdev
> - fix docs build (doxy-api.conf.in and doxy-api-index.md)
> - restore blank line in MAINTAINERS file
> - move dlb announcement in release_20_11.rst after ethdev
> - use headers = files() for exported meson public headers
> - fix a typo in 'add documentation ..." commit message
> - use eal version of cldemote
> - convert a couple of printfs to LOG messages
> - fix missing "~" in dlb documentation
> - delay introduction of _delayed token pop functions to
>   token pop commit (fixes 8 or so unused function errors)
> - all patches build incrementally (gcc), and checkpatches reports
>   success
> - I am not able to run clang locally. If clang errors are still
>   present I will ask IT to install clang on a build server tomorrow.
>
> Major changes in V14
> ====================
> - Fixed format errors in doc/api/doxy-api-index.md
> - Delayed introduction of dlb2_consume_qe_immediate until
>   add-dequeue-and-its-burst-variants.patch
> - Delayed introduction of dlb2_construct_token_pop_qe until
>   add-PMD-s-token-pop-public-interface.patch
> - Delayed introduction of dlb_equeue_*_delayed until
>   add dequeue and its burst variants.patch
>
> Major changes in V13
> ====================
> - removed now unused functions dlb_umwait and dlb_umonitor
>
> Major changes in V12
> ====================
> - Fix CENTOS build error: use __m128i instead of __v2di with
>   _mm_stream_si128
>
> Major changes in V11
> ====================
> - removed unused function, fixing build error
> - fixed typo in port_setup commit message
> - this patch series is based on dpdk-next-eventdev
>
> Major changes in v10
> =====================
> - convert to use rte_power_monitor patches
> - replace __builtin_ia32_movntdq() with _mm_stream_si128()
> - remove unused functions in dlb_selftest.c
>
> Major changes in v9
> =====================
> - fixed a build error due to __rte_cache_aligned being placed after
>   the ";" character, instead of before it.
>
> Major changes in v8 after dpdk reviews
> =====================
> - moved introduction of dlb in relnotes_20_11 to first patch in series
> - fixed underlines in dlb.rst that were too short
> - note that the code still uses its private byte-encoded versions of
>   umonitor/umwait, rather than the new functions in the power
>   patch that are built on top of those intrinsics. This is intentional.
>
> Major changes in v7 after dpdk reviews
> =====================
> - updated MAINTAINERS file to alphabetically insert DLB
> - don't create RTE_ symbols in PMD
> - converted to use version.map scheme
> - converted to use .._master_lcore instead of .._main_lcore
> - this patch set is based on dpdk-next-eventdev
>
> Major changes in v6 after dpdk reviews:
> =====================
> - fixed meson conditional build. Moved test into driver’s meson.build
>   file instead of event/meson.build
> - documentation is populated as associated code is introduced
> - add log_register in add dynamic logging patch
> - rename RTE_xxx symbol(s) as DLB2_xxx
> - replaced function ptr enqueue_four with direct call to movdir64b
> - remove unused port_pages
> - broke up probe patch into 3 smaller patches for easier review
> - changed param order of movdir64b/movntdq to match intrinsics
> - added self to MAINTAINERS files
> - squashed announcement of availability into last patch in series
> - correct spelling errors and delete repeated words
> - DPDK_21.0 -> DPDK 21 in map file
> - add experimental banner to public structs and APIs
> - implemented other suggestions from code reviews of DLB2 PMD. The
>   software is very similar in form so some DLB2 reviews comments
>   were applicable to DLB as well
>
> Major changes in v5 after dpdk reviews and additional internal reviews
> by colleagues at Intel:
> ================
> - implement changes requested in code reviews by Gage Eads and Mike Chen
> - fix a memzone leak
> - convert to use eal rte-cpuflags patch from Liang Ma
>
> Major changes in v4 after dpdk reviews and additional internal reviews
> by colleagues at Intel:
> ================
> - Remove make infrastructure
> - shared code (pf/base) is now added incrementally
> - flexible interface (iface.[ch]) is now added incrementally
> - removed calls to rte_panic
> - do not call pthread_create directly
> - remove unused internal API, os_time
> - convert rte_atomic to __atomic builtins
> - broke out eventdev ABI changes, test/api changes, and new internal PCI
>   named probe API
> - relocated enqueue logic to enqueue patch
>
> Major Changes in V3:
> ================
> - Fixed a memory corruption issue due to not allocating enough CQ
> memory for depths < 8. Hardware requires minimum allocation to be
> at least 8 entries.
> - Address review comments from Gage and Mattias.
> - Remove versioning
> - minor formatting changes
>
> Major changes in V2:
> ================
> - Correct ABI break that was present in V1.
> - Address some of the review comments received from Mattias.
>   I will address the remaining items identified by Mattias in the next
>   patch delivery.
> - General code cleanup based on internal code reviews
>
> Depends-on: patch-82202 ("eventdev: increase MAX QUEUES PER DEV to 255")
>
> Timothy McDaniel (23):
>   event/dlb: add documentation and meson infrastructure
>   event/dlb: add dynamic logging
>   event/dlb: add private data structures and constants
>   event/dlb: add definitions shared with LKM or shared code
>   event/dlb: add inline functions
>   event/dlb: add eventdev probe
>   event/dlb: add flexible interface
>   event/dlb: add probe-time hardware init
>   event/dlb: add xstats
>   event/dlb: add infos get and configure
>   event/dlb: add queue and port default conf
>   event/dlb: add queue setup
>   event/dlb: add port setup
>   event/dlb: add port link
>   event/dlb: add port unlink and port unlinks in progress
>   event/dlb: add eventdev start
>   event/dlb: add enqueue and its burst variants
>   event/dlb: add dequeue and its burst variants
>   event/dlb: add eventdev stop and close
>   event/dlb: add PMD's token pop public interface
>   event/dlb: add PMD self-tests
>   event/dlb: add queue and port release
>   event/dlb: add timeout ticks entry point
>
>  MAINTAINERS                                  |    5 +
>  app/test/test_eventdev.c                     |    7 +
>  config/rte_config.h                          |    6 +
>  doc/api/doxy-api-index.md                    |    3 +-
>  doc/api/doxy-api.conf.in                     |    1 +
>  doc/guides/eventdevs/dlb.rst                 |  341 ++
>  doc/guides/eventdevs/index.rst               |    1 +
>  doc/guides/rel_notes/release_20_11.rst       |    5 +
>  drivers/event/dlb/dlb.c                      | 4079 +++++++++++++++
>  drivers/event/dlb/dlb_iface.c                |   79 +
>  drivers/event/dlb/dlb_iface.h                |   82 +
>  drivers/event/dlb/dlb_inline_fns.h           |   36 +
>  drivers/event/dlb/dlb_log.h                  |   25 +
>  drivers/event/dlb/dlb_priv.h                 |  513 ++
>  drivers/event/dlb/dlb_selftest.c             | 1539 ++++++
>  drivers/event/dlb/dlb_user.h                 |  814 +++
>  drivers/event/dlb/dlb_xstats.c               | 1217 +++++
>  drivers/event/dlb/meson.build                |   22 +
>  drivers/event/dlb/pf/base/dlb_hw_types.h     |  334 ++
>  drivers/event/dlb/pf/base/dlb_osdep.h        |  310 ++
>  drivers/event/dlb/pf/base/dlb_osdep_bitmap.h |  441 ++
>  drivers/event/dlb/pf/base/dlb_osdep_list.h   |  131 +
>  drivers/event/dlb/pf/base/dlb_osdep_types.h  |   31 +
>  drivers/event/dlb/pf/base/dlb_regs.h         | 2368 +++++++++
>  drivers/event/dlb/pf/base/dlb_resource.c     | 6904 ++++++++++++++++++++++++++
>  drivers/event/dlb/pf/base/dlb_resource.h     |  876 ++++
>  drivers/event/dlb/pf/dlb_main.c              |  586 +++
>  drivers/event/dlb/pf/dlb_main.h              |   47 +
>  drivers/event/dlb/pf/dlb_pf.c                |  750 +++
>  drivers/event/dlb/rte_pmd_dlb.c              |   38 +
>  drivers/event/dlb/rte_pmd_dlb.h              |   77 +
>  drivers/event/dlb/version.map                |    9 +
>  drivers/event/meson.build                    |    2 +-
>  33 files changed, 21677 insertions(+), 2 deletions(-)
>  create mode 100644 doc/guides/eventdevs/dlb.rst
>  create mode 100644 drivers/event/dlb/dlb.c
>  create mode 100644 drivers/event/dlb/dlb_iface.c
>  create mode 100644 drivers/event/dlb/dlb_iface.h
>  create mode 100644 drivers/event/dlb/dlb_inline_fns.h
>  create mode 100644 drivers/event/dlb/dlb_log.h
>  create mode 100644 drivers/event/dlb/dlb_priv.h
>  create mode 100644 drivers/event/dlb/dlb_selftest.c
>  create mode 100644 drivers/event/dlb/dlb_user.h
>  create mode 100644 drivers/event/dlb/dlb_xstats.c
>  create mode 100644 drivers/event/dlb/meson.build
>  create mode 100644 drivers/event/dlb/pf/base/dlb_hw_types.h
>  create mode 100644 drivers/event/dlb/pf/base/dlb_osdep.h
>  create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_bitmap.h
>  create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_list.h
>  create mode 100644 drivers/event/dlb/pf/base/dlb_osdep_types.h
>  create mode 100644 drivers/event/dlb/pf/base/dlb_regs.h
>  create mode 100644 drivers/event/dlb/pf/base/dlb_resource.c
>  create mode 100644 drivers/event/dlb/pf/base/dlb_resource.h
>  create mode 100644 drivers/event/dlb/pf/dlb_main.c
>  create mode 100644 drivers/event/dlb/pf/dlb_main.h
>  create mode 100644 drivers/event/dlb/pf/dlb_pf.c
>  create mode 100644 drivers/event/dlb/rte_pmd_dlb.c
>  create mode 100644 drivers/event/dlb/rte_pmd_dlb.h
>  create mode 100644 drivers/event/dlb/version.map
>
> --
> 2.6.4
>