[PATCH V6 00/17] pipeline: pipeline configuration and build improvements

[Cristian Dumitrescu <cristian.dumitrescu@intel.com>]

The sequence of steps to build a SWX pipeline object are:

1. Create the specification file (pipeline.p4 -> pipeline.spec): This text file represents the
   pipeline program that is typically generated by the P4 compiler or sometimes manually created.

2. Generate the C source code file (pipeline.spec -> pipeline.c): The C file contains a C function
   for every pipeline action and several C functions for the pipeline main program. It allows the C
   compiler to generate optimized code by having access to the entire pipeline program rather than
   using the small pre-build functions corresponding to the individual instructions

3. Build the shared object library (pipeline.c -> pipeline.o -> pipeline.so).

4. Load the shared object library (pipeline.so): At initialization, the pipeline object is “patched”
   with the optimized C functions from the shared object library.

Previously, steps 2., 3. and 4. were implemented under the hood by the pipeline library at the
initialization time in a completely hard-coded and non-customizable way. The user was not able to
select the C compiler (GCC was assumed), the compiler version, the build flags, the file locations,
etc. The code generation (step 2.) and library build (step 3.) were done on-the-fly at init and
could potentially fail for many setup related reasons.

Now, this process is no longer done under the hood by the pipeline library and the individual steps
are explicitly supported by the API functions. The code generation (step 2.) is done off-line at any
time before the pipeline execution. The library build (step 3.) is also done off-line and is now
fully customizable by the user, who is able to gracefully access the generated C code and decide on
the various build options.

We also take the opportunity to streamline the pipeline I/O port configuration by introducing an I/O
specification file. Essentially, the pipeline is now configured and build based on two files:

a) The shared object library file (pipeline.so): It defines how the packets are processed by the
   program through tables and actions; the same P4 program can be executed by many pipelines.

b) The I/O specification file (pipeline.io): High level text file defining how the packets are
   received and transmitted by the pipeline; this is not part of the P4 program, which is completely
   agnostic about the pipeline I/O ports. This is defined differently for each pipeline object when
   initialized.

Change log:

V6:
-Added the cover letter. No code changes.

V5:
-Fixed more style issues reported by the CI/CD.

V4:
-Fixed style issues reported by the CI/CD.

V3:
-Added support for the pipeline I/O specification file.

V2:
-Fixed style issues reported by the CI/CD.

Cristian Dumitrescu (17):
  pipeline: add pipeline name
  pipeline: move specification data structures to internal header
  pipeline: add pipeline specification data structure
  pipeline: rework the specification file-based pipeline build
  pipeline: generate the code for pipeline specification structure
  pipeline: add support for pipeline I/O specification
  pipeline: add API for pipeline code generation
  pipeline: add API for shared library-based pipeline build
  examples/pipeline: add CLI command for pipeline code generation
  examples/pipeline: add CLI command for shared library build
  examples/pipeline: remove the obsolete pipeline create CLI command
  examples/pipeline: remove the obsolete port configuration CLI commands
  examples/pipeline: remove the obsolete mirroring configuration CLI
    command
  examples/pipeline: use the pipeline name query API
  examples/pipeline: rework the link CLI command
  examples/pipelines: remove obsolete tap CLI command
  examples/pipeline: call the code generation and build CLI commands

 examples/pipeline/cli.c                       | 1133 +++------
 examples/pipeline/examples/ethdev.io          |   27 +
 examples/pipeline/examples/fib.cli            |   44 +-
 examples/pipeline/examples/hash_func.cli      |   41 +-
 examples/pipeline/examples/l2fwd.cli          |   44 +-
 examples/pipeline/examples/l2fwd_macswp.cli   |   44 +-
 .../pipeline/examples/l2fwd_macswp_pcap.cli   |   35 +-
 examples/pipeline/examples/l2fwd_pcap.cli     |   35 +-
 examples/pipeline/examples/learner.cli        |   43 +-
 examples/pipeline/examples/meter.cli          |   58 +-
 examples/pipeline/examples/mirroring.cli      |   46 +-
 examples/pipeline/examples/pcap.io            |   27 +
 examples/pipeline/examples/recirculation.cli  |   41 +-
 examples/pipeline/examples/registers.cli      |   53 +-
 examples/pipeline/examples/selector.cli       |   55 +-
 examples/pipeline/examples/varbit.cli         |   41 +-
 examples/pipeline/examples/vxlan.cli          |   48 +-
 examples/pipeline/examples/vxlan_pcap.cli     |   39 +-
 examples/pipeline/obj.c                       |  172 +-
 examples/pipeline/obj.h                       |   46 -
 examples/pipeline/thread.c                    |   65 +-
 examples/pipeline/thread.h                    |    9 +-
 lib/pipeline/rte_swx_ctl.c                    |   99 +
 lib/pipeline/rte_swx_ctl.h                    |   15 +
 lib/pipeline/rte_swx_pipeline.c               |  464 ++--
 lib/pipeline/rte_swx_pipeline.h               |   71 +-
 lib/pipeline/rte_swx_pipeline_internal.h      |    2 +
 lib/pipeline/rte_swx_pipeline_spec.c          | 2050 +++++++++++++++--
 lib/pipeline/rte_swx_pipeline_spec.h          |  280 +++
 lib/pipeline/version.map                      |    7 +-
 30 files changed, 3430 insertions(+), 1704 deletions(-)
 create mode 100644 examples/pipeline/examples/ethdev.io
 create mode 100644 examples/pipeline/examples/pcap.io
 create mode 100644 lib/pipeline/rte_swx_pipeline_spec.h

-- 
2.34.1