[dpdk-dev] [PATCH V2] doc: add the SWX pipeline type to the prog_guide

[dpdk-dev] [PATCH V2] doc: add the SWX pipeline type to the prog_guide

[Cristian Dumitrescu]

Add the new SWX pipeline type to the Programmer's Guide.

V2 changes: fixed 2 typos.

Signed-off-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>
---
 doc/guides/prog_guide/packet_framework.rst | 77 ++++++++++++++++++++++
 1 file changed, 77 insertions(+)

diff --git a/doc/guides/prog_guide/packet_framework.rst b/doc/guides/prog_guide/packet_framework.rst
index 48d257501..3d4e3b66c 100644
--- a/doc/guides/prog_guide/packet_framework.rst
+++ b/doc/guides/prog_guide/packet_framework.rst
@@ -1148,3 +1148,80 @@ Usually, to support a specific functional block, specific implementation of Pack
 with all the implementations sharing the same API: pure SW implementation (no acceleration), implementation using accelerator A, implementation using accelerator B, etc.
 The selection between these implementations could be done at build time or at run-time (recommended), based on which accelerators are present in the system,
 with no application changes required.
+
+The Software Switch (SWX) Pipeline
+----------------------------------
+
+The Software Switch (SWX) pipeline is designed to combine the DPDK performance with the flexibility of the P4-16 language [1]. It can be used either by itself
+to code a complete software switch or data plane application, or in combination with the open-source P4 compiler P4C [2], acting as a P4C back-end that allows
+the P4 programs to be translated to the DPDK SWX API and run on multi-core CPUs.
+
+The main features of the SWX pipeline are:
+
+*   Nothing is hard-wired, everything is dynamically defined: The packet headers (i.e. the network protocols), the packet meta-data, the actions, the tables
+    and the pipeline itself are dynamically defined instead of selected from a predefined set.
+
+*   Instructions: The actions and the life of the packet through the pipeline are defined with instructions that manipulate the pipeline objects mentioned
+    above. The pipeline is the main function of the packet program, with actions as subroutines triggered by the tables.
+
+*   Call external plugins: Extern objects and functions can be defined to call functionality that cannot be efficiently implemented with the existing
+    pipeline-oriented instruction set, such as: error detecting/correcting codes, cryptographic operations, meters, statistics counter arrays, heuristics, etc.
+
+*   Better control plane interaction: Transaction-oriented table update mechanism that supports multi-table atomic updates. Multiple tables can be updated in a
+    single step with only the before-update and the after-update table entries visible to the packets. Alignment with the P4Runtime [3] protocol.
+
+*   Performance: Multiple packets are in-flight within the pipeline at any moment. Each packet is owned by a different time-sharing thread in
+    run-to-completion, with the thread pausing before memory access operations such as packet I/O and table lookup to allow the memory prefetch to complete.
+    The instructions are verified and translated at initialization time with no run-time impact. The instructions are also optimized to detect and "fuse"
+    frequently used patterns into vector-like instructions transparently to the user.
+
+The main SWX pipeline components are:
+
+*   Input and output ports: Each port instantiates a port type that defines the port operations, e.g. Ethernet device port, PCAP port, etc. The RX interface
+    of the input ports and the TX interface of the output ports are single packet based, with packet batching typically implemented internally by each port for
+    performance reasons.
+
+*   Structure types: Each structure type is used to define the logical layout of a memory block, such as: packet headers, packet meta-data, action data stored
+    in a table entry, mailboxes of extern objects and functions. Similar to C language structs, each structure type is a well defined sequence of fields, with
+    each field having a unique name and a constant size.
+
+*   Packet headers: Each packet typically has one or multiple headers. The headers are extracted from the input packet as part of the packet parsing operation,
+    which is likely executed immediately after the packet reception. As result of the extract operation, each header is logically removed from the packet, so
+    once the packet parsing operation is completed, the input packet is reduced to opaque payload. Just before transmission, one or several headers are pushed
+    in front of each output packet through the emit operation; these headers can be part of the set of headers that were previously extracted from the input
+    packet (and potentially modified afterwards) or some new headers whose contents is generated by the pipeline (e.g. by reading them from tables). The format
+    of each packet header is defined by instantiating a structure type.
+
+*   Packet meta-data: The packet meta-data is filled in by the pipeline (e.g. by reading it from tables) or computed by the pipeline. It is not sent out unless
+    some of the meta-data fields are explicitly written into the headers emitted into the output packet. The format of the packet meta-data is defined by
+    instantiating a structure type.
+
+*   Extern objects and functions: Used to plug into the pipeline any functionality that cannot be efficiently implemented with the existing pipeline instruction
+    set. Each extern object and extern function has its own mailbox, which is used to pass the input arguments to and retrieve the output arguments from the
+    extern object member functions or the extern function.  The mailbox format is defined by instantiating a structure type.
+
+*   Instructions: The pipeline and its actions are defined with instructions from a predefined instruction set. The instructions are used to receive and
+    transmit the current packet, extract and emit headers from/into the packet, read/write the packet headers, packet meta-data and mailboxes, start table
+    lookup operations, read the action arguments from the table entry, call extern object member functions or extern functions. See the rte_swx_pipeline.h file
+    for the complete list of instructions.
+
+*   Actions: The pipeline actions are dynamically defined through instructions as opposed to predefined. Essentially, the actions are subroutines of the
+    pipeline program and their execution is triggered by the table lookup. The input arguments of each action are read from the table entry (in case of table
+    lookup hit) or the default table action (in case of table lookup miss) and are read-only; their format is defined by instantiating a structure type. The
+    actions have read-write access to the packet headers and meta-data.
+
+*   Table: Each pipeline typically has one or more lookup tables. The match fields of each table are flexibly selected from the packet headers and meta-data
+    defined for the current pipeline. The set of table actions is flexibly selected for each table from the set of actions defined for the current pipeline. The
+    tables can be looked at as special pipeline operators that result in one of the table actions being called, depending on the result of the table lookup
+    operation.
+
+*   Pipeline: The pipeline represents the main program that defines the life of the packet, with subroutines (actions) executed on table lookup. As packets
+    go through the pipeline, the packet headers and meta-data are transformed along the way.
+
+References:
+
+[1] P4-16 specification: https://p4.org/specs/
+
+[2] P4-16 compiler: https://github.com/p4lang/p4c
+
+[3] P4Runtime specification: https://p4.org/specs/
-- 
2.17.1

Re: [dpdk-dev] [PATCH V2] doc: add the SWX pipeline type to the prog_guide

[David Marchand]

On Tue, Nov 10, 2020 at 9:03 PM Cristian Dumitrescu
<cristian.dumitrescu@intel.com> wrote:
>
> Add the new SWX pipeline type to the Programmer's Guide.

Thanks for this high level description.
Could you work on documenting how to start and use the new pipeline example too?

>
> V2 changes: fixed 2 typos.

Patch changelog should be an annotation.

>
> Signed-off-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>

Applied, thanks.


-- 
David Marchand

Re: [dpdk-dev] [PATCH V2] doc: add the SWX pipeline type to the prog_guide

[Dumitrescu, Cristian]

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Friday, November 13, 2020 1:04 PM
> To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>
> Cc: dev <dev@dpdk.org>
> Subject: Re: [PATCH V2] doc: add the SWX pipeline type to the prog_guide
> 
> On Tue, Nov 10, 2020 at 9:03 PM Cristian Dumitrescu
> <cristian.dumitrescu@intel.com> wrote:
> >
> > Add the new SWX pipeline type to the Programmer's Guide.
> 
> Thanks for this high level description.
> Could you work on documenting how to start and use the new pipeline
> example too?
> 

Yes, point taken. Do we still have time for 20.11 for this one?

> >
> > V2 changes: fixed 2 typos.
> 
> Patch changelog should be an annotation.
> 
> >
> > Signed-off-by: Cristian Dumitrescu <cristian.dumitrescu@intel.com>
> 
> Applied, thanks.
> 
> 
> --
> David Marchand

Re: [dpdk-dev] [PATCH V2] doc: add the SWX pipeline type to the prog_guide

[David Marchand]

On Fri, Nov 13, 2020 at 4:17 PM Dumitrescu, Cristian
<cristian.dumitrescu@intel.com> wrote:
> > -----Original Message-----
> > From: David Marchand <david.marchand@redhat.com>
> > Sent: Friday, November 13, 2020 1:04 PM
> > To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>
> > Cc: dev <dev@dpdk.org>
> > Subject: Re: [PATCH V2] doc: add the SWX pipeline type to the prog_guide
> >
> > On Tue, Nov 10, 2020 at 9:03 PM Cristian Dumitrescu
> > <cristian.dumitrescu@intel.com> wrote:
> > >
> > > Add the new SWX pipeline type to the Programmer's Guide.
> >
> > Thanks for this high level description.
> > Could you work on documenting how to start and use the new pipeline
> > example too?
> >
>
> Yes, point taken. Do we still have time for 20.11 for this one?

For documentation, yes.


-- 
David Marchand

Re: [dpdk-dev] [PATCH V2] doc: add the SWX pipeline type to the prog_guide

[David Marchand]

On Sun, Nov 15, 2020 at 4:58 PM David Marchand
<david.marchand@redhat.com> wrote:
>
> On Fri, Nov 13, 2020 at 4:17 PM Dumitrescu, Cristian
> <cristian.dumitrescu@intel.com> wrote:
> > > -----Original Message-----
> > > From: David Marchand <david.marchand@redhat.com>
> > > Sent: Friday, November 13, 2020 1:04 PM
> > > To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>
> > > Cc: dev <dev@dpdk.org>
> > > Subject: Re: [PATCH V2] doc: add the SWX pipeline type to the prog_guide
> > >
> > > On Tue, Nov 10, 2020 at 9:03 PM Cristian Dumitrescu
> > > <cristian.dumitrescu@intel.com> wrote:
> > > >
> > > > Add the new SWX pipeline type to the Programmer's Guide.
> > >
> > > Thanks for this high level description.
> > > Could you work on documenting how to start and use the new pipeline
> > > example too?
> > >
> >
> > Yes, point taken. Do we still have time for 20.11 for this one?
>
> For documentation, yes.

ping.


-- 
David Marchand

Re: [dpdk-dev] [PATCH V2] doc: add the SWX pipeline type to the prog_guide

[Dumitrescu, Cristian]

> -----Original Message-----
> From: David Marchand <david.marchand@redhat.com>
> Sent: Monday, November 23, 2020 4:10 PM
> To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>
> Cc: dev <dev@dpdk.org>; Thomas Monjalon <thomas@monjalon.net>
> Subject: Re: [PATCH V2] doc: add the SWX pipeline type to the prog_guide
> 
> On Sun, Nov 15, 2020 at 4:58 PM David Marchand
> <david.marchand@redhat.com> wrote:
> >
> > On Fri, Nov 13, 2020 at 4:17 PM Dumitrescu, Cristian
> > <cristian.dumitrescu@intel.com> wrote:
> > > > -----Original Message-----
> > > > From: David Marchand <david.marchand@redhat.com>
> > > > Sent: Friday, November 13, 2020 1:04 PM
> > > > To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>
> > > > Cc: dev <dev@dpdk.org>
> > > > Subject: Re: [PATCH V2] doc: add the SWX pipeline type to the
> prog_guide
> > > >
> > > > On Tue, Nov 10, 2020 at 9:03 PM Cristian Dumitrescu
> > > > <cristian.dumitrescu@intel.com> wrote:
> > > > >
> > > > > Add the new SWX pipeline type to the Programmer's Guide.
> > > >
> > > > Thanks for this high level description.
> > > > Could you work on documenting how to start and use the new pipeline
> > > > example too?
> > > >
> > >
> > > Yes, point taken. Do we still have time for 20.11 for this one?
> >
> > For documentation, yes.
> 
> ping.
> 
> 
> --
> David Marchand

Hi David,

Done: http://patches.dpdk.org/patch/84475/

Thanks,
Cristian