[RFC v3 0/3] Add event dispatcher

["Mattias Rönnblom" <mattias.ronnblom@ericsson.com>]

The purpose of the event dispatcher is to decouple different parts of
an application (e.g., processing pipeline stages), sharing the same
underlying event device.

The event dispatcher replaces the conditional logic (often, a switch
statement) that typically follows an event device dequeue operation,
where events are dispatched to different parts of the application
based on event meta data, such as the queue id or scheduling type.

The concept is similar to a UNIX file descriptor event loop library.
Instead of tying callback functions to fds as for example libevent
does, the event dispatcher relies on application-supplied matching
callback functions to decide where to deliver events.

An event dispatcher is configured to dequeue events from a specific
event device, and ties into the service core framework, to do its (and
the application's) work.

The event dispatcher provides a convenient way for an eventdev-based
application to use service cores for application-level processing, and
thus for sharing those cores with other DPDK services.

Although the event dispatcher adds some overhead, experience suggests
that the net effect on the application (both syntetic benchmarks and
more real-world applications) seems to be positive, except for small
dequeue sizes. This is likely due to improved temporal locality, and
will very greatly. The overhead seems to be ~10 cc/event on a large
core, with a handful of handlers.

Outstanding questions:
 o Explore an option where the callbacks (and callback data) are per lcore.
   More complicated, but more performant for deployments where all
   dispatcher lcores are not used for the same purpose.
 o Reshuffle the handler order in runtime, in such a way that the
   most-often-matched handler is tried first.
 o Consider adding possibility to express simple match functions
   (e..queue_id == 7) without a match callback.
 o Consider runtime reconfiguration.
 o Consider moving dispatcher id allocation from app to DPDK. Avoids
   app-level coordination, but makes the API inconsistent with other
   Eventdev APIs.
 o Should the events delivered to the application in the process callback
   be marked 'const' or not ('const' now, but prohibits reuse for TX).
 o Consider allowing the application setting the process callback to NULL,
   signalling to the dispatcher that processing will occur already at the
   time of the match call. May provide some slight performance benefits
   for applications where the averagenumber of events supplied per process
   function call is very small.

RFC v3:
 o Change from int16_t to int for ids. No particular gain in using
   int16_t over int.
 o Introduce rte_event_dispatcher_start()/stop() to make API consistent
   with the other Eventdev services.
 o Unit tests added.
 o Programming guide added.
 o Added dispatcher statistics.
 o Abandonded the promise to run the match functions in order. This is
   to allow for future performance optimization (i.e., reorder match
   functions so that often-matched functions are checked first).
 o Introduced event_port_id and event_dev_id in the deliver
   callback, to simplify for applications to follow the Eventdev
   API requirements to enqueue forward-type events on the same
   event port as the original events were dequeued.
 o Mark delivered events as constant.
 o Introduce optional callback called by the dispatcher when a full
   batch of events has been distributed to the various handlers. This
   is useful in cases when an event output buffer is shared among
   several handlers.
 o Fixed bug where consumer unregistration would invalidate other
   registration ids.

RFC v2:
 o Introduced a match callback, allowing the application layer to
   route events to callbacks not only based on queue id (like in v1),
   but on arbitrary event meta data or payload.
 o The fallback was removed, since it now easily can be achieved with a
   match function always returning true.
 o The dispatcher now rearranges the batch of dequeued events from the
   event device, in such a way to minimize the number of deliver calls
   made. This is done primariy to improved application cache locality.

Mattias Rönnblom (3):
  eventdev: introduce event dispatcher
  test: add event dispatcher test suite
  doc: add event dispatcher programming guide

 app/test/meson.build                       |   1 +
 app/test/test_event_dispatcher.c           | 814 +++++++++++++++++++++
 doc/api/doxy-api-index.md                  |   1 +
 doc/guides/prog_guide/event_dispatcher.rst | 423 +++++++++++
 doc/guides/prog_guide/index.rst            |   1 +
 lib/eventdev/meson.build                   |   2 +
 lib/eventdev/rte_event_dispatcher.c        | 670 +++++++++++++++++
 lib/eventdev/rte_event_dispatcher.h        | 440 +++++++++++
 lib/eventdev/version.map                   |  12 +
 9 files changed, 2364 insertions(+)
 create mode 100644 app/test/test_event_dispatcher.c
 create mode 100644 doc/guides/prog_guide/event_dispatcher.rst
 create mode 100644 lib/eventdev/rte_event_dispatcher.c
 create mode 100644 lib/eventdev/rte_event_dispatcher.h

-- 
2.34.1