* [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide
@ 2017-03-15 17:21 Harry van Haaren
2017-03-30 9:14 ` Burakov, Anatoly
2017-03-30 13:36 ` Jerin Jacob
0 siblings, 2 replies; 6+ messages in thread
From: Harry van Haaren @ 2017-03-15 17:21 UTC (permalink / raw)
To: dev; +Cc: Harry van Haaren
This commit adds an entry in the programmers guide
explaining the eventdev library.
The rte_event struct, queues and ports are explained.
An API walktrough of a simple two stage atomic pipeline
provides the reader with a step by step overview of the
expected usage of the Eventdev API.
Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
---
There are checkpatch whitespace warnings reported; they're
part of the exported SVG image.
---
doc/guides/prog_guide/eventdev.rst | 360 ++++++++++
doc/guides/prog_guide/img/eventdev_usage.svg | 994 +++++++++++++++++++++++++++
doc/guides/prog_guide/index.rst | 1 +
3 files changed, 1355 insertions(+)
create mode 100644 doc/guides/prog_guide/eventdev.rst
create mode 100644 doc/guides/prog_guide/img/eventdev_usage.svg
diff --git a/doc/guides/prog_guide/eventdev.rst b/doc/guides/prog_guide/eventdev.rst
new file mode 100644
index 0000000..802a425
--- /dev/null
+++ b/doc/guides/prog_guide/eventdev.rst
@@ -0,0 +1,360 @@
+.. BSD LICENSE
+ Copyright(c) 2017 Intel Corporation. All rights reserved.
+
+ Redistribution and use in source and binary forms, with or without
+ modification, are permitted provided that the following conditions
+ are met:
+
+ * Redistributions of source code must retain the above copyright
+ notice, this list of conditions and the following disclaimer.
+ * Redistributions in binary form must reproduce the above copyright
+ notice, this list of conditions and the following disclaimer in
+ the documentation and/or other materials provided with the
+ distribution.
+ * Neither the name of Intel Corporation nor the names of its
+ contributors may be used to endorse or promote products derived
+ from this software without specific prior written permission.
+
+ THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+Event Device Library
+====================
+
+The DPDK Event device library is an abstraction that provides the application
+with features to schedule events. This is achieved using the PMD architecture
+similar to the ethdev or cryptodev APIs, which may already be familiar to the
+reader. The eventdev framework is provided as a DPDK library, allowing
+applications to use it if they wish, but not require its usage.
+
+The goal of this library is to enable applications to build processing
+pipelines where the load balancing and scheduling is handled by the eventdev.
+Step-by-step instructions of the eventdev design is available in the `API
+Walktrough`_ section later in this document.
+
+Event struct
+------------
+
+The eventdev API represents each event with a generic struct, which contains a
+payload and metadata required for scheduling by an eventdev. The
+``rte_event`` struct is a 16 byte C structure, defined in
+``libs/librte_eventdev/rte_eventdev.h``.
+
+Event Metadata
+~~~~~~~~~~~~~~
+
+The rte_event structure contains the following metadata fields, which the
+application fills in to have the event scheduled as required:
+
+* ``flow_id`` - The targeted flow identifier for the enq/deq operation.
+* ``event_type`` - The source of this event, eg RTE_EVENT_TYPE_ETHDEV or CPU.
+* ``sub_event_type`` - Distinguishes events inside the application, that have
+ the same event_type (see above)
+* ``op`` - This field takes one of the RTE_EVENT_OP_* values, and tells the
+ eventdev about the status of the event - valid values are NEW, FORWARD or
+ RELEASE.
+* ``sched_type`` - Represents the type of scheduling that should be performed
+ on this event, valid values are the RTE_SCHED_TYPE_ORDERED, ATOMIC and
+ PARALLEL.
+* ``queue_id`` - The identifier for the event queue that the event is sent to.
+* ``priority`` - The priority of this event, see RTE_EVENT_DEV_PRIORITY.
+
+Event Payload
+~~~~~~~~~~~~~
+
+The rte_event struct contains a union for payload, allowing flexibility in what
+the actual event being scheduled is. The payload is a union of the following:
+
+* ``uint64_t u64``
+* ``void *event_ptr``
+* ``struct rte_mbuf *mbuf``
+
+These three items in a union occupy the same 64 bits at the end of the rte_event
+structure. The application can utilize the 64 bits directly by accessing the
+u64 variable, while the event_ptr and mbuf are provided as convenience
+variables. For example the mbuf pointer in the union can used to schedule a
+DPDK packet.
+
+Queues
+~~~~~~
+
+A queue is a logical "stage" of a packet processing graph, where each stage
+has a specified scheduling type. The application configures each queue for a
+specific type of scheduling, and just enqueues all events to the eventdev.
+The Eventdev API supports the following scheduling types per queue:
+
+* Atomic
+* Ordered
+* Parallel
+* Single-Link
+
+Atomic, Ordered and Parallel are load-balanced scheduling types: the output
+of the queue can be spread out over multiple CPU cores. Single-link is a
+unique queue type, which indicates the output is consumed by a single core.
+
+Atomic scheduling on a queue ensures that a single flow is not present on two
+different CPU cores at the same time. Ordered allows sending all flows to any
+core, but the scheduler must ensure that on egress the packets are returned to
+ingress order. Parallel allows sending all flows to all CPU cores, without any
+re-ordering guarantees. Single-link follows a FIFO like structure, maintaining
+ordering but it is only capable of being linked to a single port (see below
+for port and queue linking details).
+
+Ports
+~~~~~
+
+Ports are the points of contact between worker cores and the eventdev. The
+general use-case will see one CPU core using one port to enqueue and dequeue
+events from an eventdev. Ports are linked to queues in order to retrieve events
+from those queues (more details in `Linking Queues and Ports`_ below).
+
+
+API Walktrough
+--------------
+
+This section will introduce the reader to the eventdev API, showing how to
+create and configure an eventdev and use it for a two-stage atomic pipeline
+with a single core for TX. The diagram below shows the final state of the
+application after this walktrough:
+
+.. _figure_eventdev-usage1:
+
+.. figure:: img/eventdev_usage.*
+
+ Sample eventdev usage, with RX, two atomic stages and a single-link to TX.
+
+
+A high level overview of the setup steps are:
+
+* rte_event_dev_configure()
+* rte_event_queue_setup()
+* rte_event_port_setup()
+* rte_event_port_link()
+* rte_event_dev_start()
+
+
+Init and Config
+~~~~~~~~~~~~~~~
+
+The eventdev library uses vdev options to add devices to the DPDK application.
+The ``--vdev`` EAL option allows adding eventdev instances to your DPDK
+application, using the name of the eventdev PMD as an argument.
+
+For example, to create an instance of the software eventdev scheduler, the
+following vdev arguments should be provided to the application EAL command line:
+
+.. code-block:: console
+
+ ./dpdk_application --vdev="event_sw0"
+
+In the following code, we configure eventdev instance with 3 queues
+and 6 ports as follows. The 3 queues consist of 2 Atomic and 1 Single-Link,
+while the 6 ports consist of 4 workers, 1 RX and 1 TX.
+
+.. code-block:: c
+
+ const struct rte_event_dev_config config = {
+ .nb_event_queues = 3,
+ .nb_event_ports = 6,
+ .nb_events_limit = 4096,
+ .nb_event_queue_flows = 1024,
+ .nb_event_port_dequeue_depth = 128,
+ .nb_event_port_enqueue_depth = 128,
+ };
+ int err = rte_event_dev_configure(dev_id, &config);
+
+The remainder of this walktrough assumes that dev_id is 0.
+
+Setting up Queues
+~~~~~~~~~~~~~~~~~
+
+Once the eventdev itself is configured, the next step is to configure queues.
+This is done by setting the appropriate values in a queue_conf structure, and
+calling the setup function. Repeat this step for each queue, starting from
+0 and ending at ``nb_event_queues - 1`` from the event_dev config above.
+
+.. code-block:: c
+
+ struct rte_event_queue_conf atomic_conf = {
+ .event_queue_cfg = RTE_EVENT_QUEUE_CFG_ATOMIC_ONLY,
+ .priority = RTE_EVENT_DEV_PRIORITY_NORMAL,
+ .nb_atomic_flows = 1024,
+ .nb_atomic_order_sequences = 1024,
+ };
+ int dev_id = 0;
+ int queue_id = 0;
+ int err = rte_event_queue_setup(dev_id, queue_id, &atomic_conf);
+
+The remainder of this walktrough assumes that the queues are configured as
+follows:
+
+ * id 0, atomic queue #1
+ * id 1, atomic queue #2
+ * id 2, single-link queue
+
+Setting up Ports
+~~~~~~~~~~~~~~~~
+
+Once queues are set up successfully, create the ports as required. Each port
+should be set up with its corresponding port_conf type, worker for worker cores,
+rx and tx for the RX and TX cores:
+
+.. code-block:: c
+
+ struct rte_event_port_conf rx_conf = {
+ .dequeue_depth = 128,
+ .enqueue_depth = 128,
+ .new_event_threshold = 1024,
+ };
+ struct rte_event_port_conf worker_conf = {
+ .dequeue_depth = 16,
+ .enqueue_depth = 64,
+ .new_event_threshold = 4096,
+ };
+ struct rte_event_port_conf tx_conf = {
+ .dequeue_depth = 128,
+ .enqueue_depth = 128,
+ .new_event_threshold = 4096,
+ };
+ int dev_id = 0;
+ int port_id = 0;
+ int err = rte_event_port_setup(dev_id, port_id, &CORE_FUNCTION_conf);
+
+It is now assumed that:
+
+ * port 0: RX core
+ * ports 1,2,3,4: Workers
+ * port 5: TX core
+
+Linking Queues and Ports
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+The final step is to "wire up" the ports to the queues. After this, the
+eventdev is capable of scheduling events, and when cores request work to do,
+the correct events are provided to that core. Note that the RX core takes input
+from eg: a NIC so it is not linked to any eventdev queues.
+
+Linking all workers to atomic queues, and the TX core to the single-link queue
+can be achieved like this:
+
+.. code-block:: c
+
+ uint8_t port_id = 0;
+ uint8_t atomic_qs[] = {0, 1};
+ uint8_t single_link_q = 2;
+ uint8_t tx_port_id = 5;
+ uin8t_t priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+
+ for(int i = 0; i < 4; i++) {
+ int worker_port = i + 1;
+ int links_made = rte_event_port_link(dev_id, worker_port, atomic_qs, NULL, 2);
+ }
+ int links_made = rte_event_port_link(dev_id, tx_port_id, &single_link_q, &priority, 1);
+
+Starting the EventDev
+~~~~~~~~~~~~~~~~~~~~~
+
+A single function call tells the eventdev instance to start processing
+events. Note that all queues must be linked to for the instance to start, as
+if any queue is not linked to, enqueuing to that queue will cause the
+application to backpressure and eventually stall due to no space in the
+eventdev.
+
+.. code-block:: c
+
+ int err = rte_event_dev_start(dev_id);
+
+Ingress of New Events
+~~~~~~~~~~~~~~~~~~~~~
+
+Now that the eventdev is set up, and ready to receive events, the RX core must
+enqueue some events into the system for it to schedule. The events to be
+scheduled are ordinary DPDK packets, received from an eth_rx_burst() as normal.
+The following code shows how those packets can be enqueued into the eventdev:
+
+.. code-block:: c
+
+ const uint16_t nb_rx = rte_eth_rx_burst(eth_port, 0, mbufs, BATCH_SIZE);
+
+ for (i = 0; i < nb_rx; i++) {
+ ev[i].flow_id = mbufs[i]->hash.rss;
+ ev[i].op = RTE_EVENT_OP_NEW;
+ ev[i].sched_type = RTE_EVENT_QUEUE_CFG_ATOMIC_ONLY;
+ ev[i].queue_id = 0;
+ ev[i].event_type = RTE_EVENT_TYPE_CPU;
+ ev[i].sub_event_type = 0;
+ ev[i].priority = RTE_EVENT_DEV_PRIORITY_NORMAL;
+ ev[i].mbuf = mbufs[i];
+ }
+
+ const int nb_tx = rte_event_enqueue_burst(dev_id, port_id, ev, nb_rx);
+ if (nb_tx != nb_rx) {
+ for(i = nb_tx; i < nb_rx; i++)
+ rte_pktmbuf_free(mbufs[i]);
+ }
+
+Forwarding of Events
+~~~~~~~~~~~~~~~~~~~~
+
+Now that the RX core has injected events, there is work to be done by the
+workers. Note that each worker will dequeue as many events as it can in a burst,
+process each one individually, and then burst the packets back into the
+eventdev.
+
+The worker can lookup the events source from ``event.queue_id``, which should
+indicate to the worker what workload needs to be performed on the event.
+Once done, the worker can update the ``event.queue_id`` to a new value, to send
+the event to the next stage in the pipeline.
+
+.. code-block:: c
+
+ int timeout = 0;
+ struct rte_event events[BATCH_SIZE];
+ uint16_t nb_rx = rte_event_dequeue_burst(dev_id, worker_port_id, events, BATCH_SIZE, timeout);
+
+ for (i = 0; i < nb_rx; i++) {
+ /* process mbuf using events[i].queue_id as pipeline stage */
+ struct rte_mbuf *mbuf = events[i].mbuf;
+ /* Send event to next stage in pipeline */
+ events[i].queue_id++;
+ }
+
+ uint16_t nb_tx = rte_event_enqueue_burst(dev_id, port_id, events, nb_rx);
+
+
+Egress of Events
+~~~~~~~~~~~~~~~~
+
+Finally, when the packet is ready for egress or needs to be dropped, we need
+to inform the eventdev that the packet is no longer being handled by the
+application. This can be done by calling dequeue() or dequeue_burst(), which
+indicates that the previous burst of packets is no longer in use by the
+application.
+
+.. code-block:: c
+
+ struct rte_event events[BATCH_SIZE];
+ uint16_t n = rte_event_dequeue_burst(dev_id, port_id, events, BATCH_SIZE, 0);
+ /* burst #1 : now tx or use the packets */
+ n = rte_event_dequeue_burst(dev_id, port_id, events, BATCH_SIZE, 0);
+ /* burst #1 is now no longer valid to use in the application, as
+ the eventdev has dropped any locks or released re-ordered packets */
+
+Summary
+-------
+
+The eventdev library allows an application to easily schedule events as it
+requires, either using a run-to-completion or pipeline processing model. The
+queues and ports abstract the logical functionality of an eventdev, providing
+the application with a generic method to schedule events. With the flexible
+PMD infrastructure applications benefit of improvements in existing eventdevs
+and additions of new ones without modification.
diff --git a/doc/guides/prog_guide/img/eventdev_usage.svg b/doc/guides/prog_guide/img/eventdev_usage.svg
new file mode 100644
index 0000000..4b660ce
--- /dev/null
+++ b/doc/guides/prog_guide/img/eventdev_usage.svg
@@ -0,0 +1,994 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+
+<svg
+ xmlns:v="http://schemas.microsoft.com/visio/2003/SVGExtensions/"
+ xmlns:dc="http://purl.org/dc/elements/1.1/"
+ xmlns:cc="http://creativecommons.org/ns#"
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ xmlns:svg="http://www.w3.org/2000/svg"
+ xmlns="http://www.w3.org/2000/svg"
+ xmlns:xlink="http://www.w3.org/1999/xlink"
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
+ width="683.12061"
+ height="184.672"
+ viewBox="0 0 546.49648 147.7376"
+ xml:space="preserve"
+ color-interpolation-filters="sRGB"
+ class="st9"
+ id="svg2"
+ version="1.1"
+ inkscape:version="0.48.4 r9939"
+ sodipodi:docname="eventdev_usage.svg"
+ style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible"><metadata
+ id="metadata214"><rdf:RDF><cc:Work
+ rdf:about=""><dc:format>image/svg+xml</dc:format><dc:type
+ rdf:resource="http://purl.org/dc/dcmitype/StillImage" /></cc:Work></rdf:RDF></metadata><sodipodi:namedview
+ pagecolor="#ffffff"
+ bordercolor="#666666"
+ borderopacity="1"
+ objecttolerance="10"
+ gridtolerance="10"
+ guidetolerance="10"
+ inkscape:pageopacity="0"
+ inkscape:pageshadow="2"
+ inkscape:window-width="1920"
+ inkscape:window-height="1017"
+ id="namedview212"
+ showgrid="false"
+ fit-margin-top="2"
+ fit-margin-left="2"
+ fit-margin-right="2"
+ fit-margin-bottom="2"
+ inkscape:zoom="1.2339869"
+ inkscape:cx="501.15554"
+ inkscape:cy="164.17693"
+ inkscape:window-x="-8"
+ inkscape:window-y="406"
+ inkscape:window-maximized="1"
+ inkscape:current-layer="g17" />
+ <v:documentProperties
+ v:langID="1033"
+ v:viewMarkup="false">
+ <v:userDefs>
+ <v:ud
+ v:nameU="msvSubprocessMaster"
+ v:prompt=""
+ v:val="VT4(Rectangle)" />
+ <v:ud
+ v:nameU="msvNoAutoConnect"
+ v:val="VT0(1):26" />
+ </v:userDefs>
+ </v:documentProperties>
+
+ <style
+ type="text/css"
+ id="style4">
+
+ .st1 {visibility:visible}
+ .st2 {fill:#5b9bd5;fill-opacity:0.22;filter:url(#filter_2);stroke:#5b9bd5;stroke-opacity:0.22}
+ .st3 {fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25}
+ .st4 {fill:#feffff;font-family:Calibri;font-size:0.833336em}
+ .st5 {font-size:1em}
+ .st6 {fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25}
+ .st7 {marker-end:url(#mrkr4-33);stroke:#5b9bd5;stroke-linecap:round;stroke-linejoin:round;stroke-width:1}
+ .st8 {fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-opacity:1;stroke-width:0.28409090909091}
+ .st9 {fill:none;fill-rule:evenodd;font-size:12px;overflow:visible;stroke-linecap:square;stroke-miterlimit:3}
+
+ </style>
+
+ <defs
+ id="Markers">
+ <g
+ id="lend4">
+ <path
+ d="M 2,1 0,0 2,-1 2,1"
+ style="stroke:none"
+ id="path8"
+ inkscape:connector-curvature="0" />
+ </g>
+ <marker
+ id="mrkr4-33"
+ class="st8"
+ v:arrowType="4"
+ v:arrowSize="2"
+ v:setback="7.04"
+ refX="-7.04"
+ orient="auto"
+ markerUnits="strokeWidth"
+ overflow="visible"
+ style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible">
+ <use
+ xlink:href="#lend4"
+ transform="scale(-3.52,-3.52)"
+ id="use11"
+ x="0"
+ y="0"
+ width="3"
+ height="3" />
+ </marker>
+ <filter
+ id="filter_2-7"
+ color-interpolation-filters="sRGB"><feGaussianBlur
+ stdDeviation="2"
+ id="feGaussianBlur15-1" /></filter><marker
+ id="mrkr4-33-2"
+ class="st8"
+ v:arrowType="4"
+ v:arrowSize="2"
+ v:setback="7.04"
+ refX="-7.04"
+ orient="auto"
+ markerUnits="strokeWidth"
+ overflow="visible"
+ style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible"><use
+ xlink:href="#lend4"
+ transform="scale(-3.52,-3.52)"
+ id="use11-3"
+ x="0"
+ y="0"
+ width="3"
+ height="3" /></marker><marker
+ id="mrkr4-33-6"
+ class="st8"
+ v:arrowType="4"
+ v:arrowSize="2"
+ v:setback="7.04"
+ refX="-7.04"
+ orient="auto"
+ markerUnits="strokeWidth"
+ overflow="visible"
+ style="fill:#5b9bd5;fill-opacity:1;stroke:#5b9bd5;stroke-width:0.28409091;stroke-opacity:1;overflow:visible"><use
+ xlink:href="#lend4"
+ transform="scale(-3.52,-3.52)"
+ id="use11-8"
+ x="0"
+ y="0"
+ width="3"
+ height="3" /></marker></defs>
+ <defs
+ id="Filters">
+ <filter
+ id="filter_2"
+ color-interpolation-filters="sRGB">
+ <feGaussianBlur
+ stdDeviation="2"
+ id="feGaussianBlur15" />
+ </filter>
+ </defs>
+ <g
+ v:mID="0"
+ v:index="1"
+ v:groupContext="foregroundPage"
+ id="g17"
+ transform="translate(-47.323579,-90.784072)">
+ <v:userDefs>
+ <v:ud
+ v:nameU="msvThemeOrder"
+ v:val="VT0(0):26" />
+ </v:userDefs>
+ <title
+ id="title19">Page-1</title>
+ <v:pageProperties
+ v:drawingScale="1"
+ v:pageScale="1"
+ v:drawingUnits="0"
+ v:shadowOffsetX="9"
+ v:shadowOffsetY="-9" />
+ <v:layer
+ v:name="Connector"
+ v:index="0" />
+ <g
+ id="shape1-1"
+ v:mID="1"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,128.62352,-288.18843)">
+ <title
+ id="title22">Square</title>
+ <desc
+ id="desc24">Atomic Queue #1</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="30.75"
+ cy="581.25"
+ width="61.5"
+ height="61.5" />
+ <g
+ id="shadow1-2"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <rect
+ x="0"
+ y="550.5"
+ width="61.5"
+ height="61.5"
+ class="st2"
+ id="rect27"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <rect
+ x="0"
+ y="550.5"
+ width="61.5"
+ height="61.5"
+ class="st3"
+ id="rect29"
+ style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" />
+
+ </g>
+ <g
+ id="shape3-8"
+ v:mID="3"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,297.37175,-288.18843)">
+ <title
+ id="title36">Square.3</title>
+ <desc
+ id="desc38">Atomic Queue #2</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="30.75"
+ cy="581.25"
+ width="61.5"
+ height="61.5" />
+ <g
+ id="shadow3-9"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <rect
+ x="0"
+ y="550.5"
+ width="61.5"
+ height="61.5"
+ class="st2"
+ id="rect41"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <rect
+ x="0"
+ y="550.5"
+ width="61.5"
+ height="61.5"
+ class="st3"
+ id="rect43"
+ style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" />
+
+ </g>
+ <g
+ id="shape4-15"
+ v:mID="4"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,466.1192,-288.18843)">
+ <title
+ id="title50">Square.4</title>
+ <desc
+ id="desc52">Single Link Queue # 1</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="30.75"
+ cy="581.25"
+ width="61.5"
+ height="61.5" />
+ <g
+ id="shadow4-16"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <rect
+ x="0"
+ y="550.5"
+ width="61.5"
+ height="61.5"
+ class="st2"
+ id="rect55"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <rect
+ x="0"
+ y="550.5"
+ width="61.5"
+ height="61.5"
+ class="st3"
+ id="rect57"
+ style="fill:#5b9bd5;stroke:#c7c8c8;stroke-width:0.25" />
+
+ </g>
+ <g
+ id="shape5-22"
+ v:mID="5"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,52.208527,-296.14701)">
+ <title
+ id="title64">Circle</title>
+ <desc
+ id="desc66">RX</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="20.5"
+ cy="591.5"
+ width="35.88"
+ height="30.75" />
+ <g
+ id="shadow5-23"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st2"
+ id="path69"
+ inkscape:connector-curvature="0"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st6"
+ id="path71"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+ <text
+ x="15.19"
+ y="594.5"
+ class="st4"
+ v:langID="1033"
+ id="text73"
+ style="fill:#feffff;font-family:Calibri"><v:paragraph
+ v:horizAlign="1" /><v:tabList />RX</text>
+
+ </g>
+ <g
+ id="shape6-28"
+ v:mID="6"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,84.042834,-305.07614)">
+ <title
+ id="title76">Dynamic connector</title>
+ <path
+ d="m 0,603 50.38,0"
+ class="st7"
+ id="path78"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape7-34"
+ v:mID="7"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,220.95621,-296.14701)">
+ <title
+ id="title81">Circle.7</title>
+ <desc
+ id="desc83">W ..</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="20.5"
+ cy="591.5"
+ width="35.88"
+ height="30.75" />
+ <g
+ id="shadow7-35"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st2"
+ id="path86"
+ inkscape:connector-curvature="0"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st6"
+ id="path88"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+ <text
+ x="12.4"
+ y="594.5"
+ class="st4"
+ v:langID="1033"
+ id="text90"
+ style="fill:#feffff;font-family:Calibri"><v:paragraph
+ v:horizAlign="1" /><v:tabList />W ..</text>
+
+ </g>
+ <g
+ id="shape9-40"
+ v:mID="9"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,220.95621,-243.34865)">
+ <title
+ id="title93">Circle.9</title>
+ <desc
+ id="desc95">W N</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="20.5"
+ cy="591.5"
+ width="35.88"
+ height="30.75" />
+ <g
+ id="shadow9-41"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st2"
+ id="path98"
+ inkscape:connector-curvature="0"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st6"
+ id="path100"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+ <text
+ x="11.69"
+ y="594.5"
+ class="st4"
+ v:langID="1033"
+ id="text102"
+ style="fill:#feffff;font-family:Calibri"><v:paragraph
+ v:horizAlign="1" /><v:tabList />W N</text>
+
+ </g>
+ <g
+ id="shape10-46"
+ v:mID="10"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,220.95621,-348.94537)">
+ <title
+ id="title105">Circle.10</title>
+ <desc
+ id="desc107">W 1</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="20.5"
+ cy="591.5"
+ width="35.88"
+ height="30.75" />
+ <g
+ id="shadow10-47"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st2"
+ id="path110"
+ inkscape:connector-curvature="0"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st6"
+ id="path112"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+ <text
+ x="12.39"
+ y="594.5"
+ class="st4"
+ v:langID="1033"
+ id="text114"
+ style="fill:#feffff;font-family:Calibri"><v:paragraph
+ v:horizAlign="1" /><v:tabList />W 1</text>
+
+ </g>
+ <g
+ id="shape11-52"
+ v:mID="11"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,195.91581,-312.06416)">
+ <title
+ id="title117">Dynamic connector.11</title>
+ <path
+ d="m 0,612 0,-68 25.21,0"
+ class="st7"
+ id="path119"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape12-57"
+ v:mID="12"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,176.37498,-305.07614)">
+ <title
+ id="title122">Dynamic connector.12</title>
+ <path
+ d="m 0,603 50.38,0"
+ class="st7"
+ id="path124"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape13-62"
+ v:mID="13"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,176.37498,-312.06416)">
+ <title
+ id="title127">Dynamic connector.13</title>
+ <path
+ d="m 0,612 25.17,0 0,68 25.21,0"
+ class="st7"
+ id="path129"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape14-67"
+ v:mID="14"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,252.79052,-259.2658)">
+ <title
+ id="title132">Dynamic connector.14</title>
+ <path
+ d="m 0,612 26.88,0 0,-68 23.5,0"
+ class="st7"
+ id="path134"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape15-72"
+ v:mID="15"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,252.79052,-305.07614)">
+ <title
+ id="title137">Dynamic connector.15</title>
+ <path
+ d="m 0,603 50.38,0"
+ class="st7"
+ id="path139"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape19-77"
+ v:mID="19"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,389.70366,-296.14701)">
+ <title
+ id="title142">Circle.19</title>
+ <desc
+ id="desc144">W ..</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="20.5"
+ cy="591.5"
+ width="35.88"
+ height="30.75" />
+ <g
+ id="shadow19-78"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st2"
+ id="path147"
+ inkscape:connector-curvature="0"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st6"
+ id="path149"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+ <text
+ x="12.4"
+ y="594.5"
+ class="st4"
+ v:langID="1033"
+ id="text151"
+ style="fill:#feffff;font-family:Calibri"><v:paragraph
+ v:horizAlign="1" /><v:tabList />W ..</text>
+
+ </g>
+ <g
+ id="shape20-83"
+ v:mID="20"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,389.70366,-243.34865)">
+ <title
+ id="title154">Circle.20</title>
+ <desc
+ id="desc156">W N</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="20.5"
+ cy="591.5"
+ width="35.88"
+ height="30.75" />
+ <g
+ id="shadow20-84"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st2"
+ id="path159"
+ inkscape:connector-curvature="0"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st6"
+ id="path161"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+ <text
+ x="11.69"
+ y="594.5"
+ class="st4"
+ v:langID="1033"
+ id="text163"
+ style="fill:#feffff;font-family:Calibri"><v:paragraph
+ v:horizAlign="1" /><v:tabList />W N</text>
+
+ </g>
+ <g
+ id="shape21-89"
+ v:mID="21"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,389.70366,-348.94537)">
+ <title
+ id="title166">Circle.21</title>
+ <desc
+ id="desc168">W 1</desc>
+ <v:userDefs>
+ <v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" />
+ </v:userDefs>
+ <v:textBlock
+ v:margins="rect(4,4,4,4)" />
+ <v:textRect
+ cx="20.5"
+ cy="591.5"
+ width="35.88"
+ height="30.75" />
+ <g
+ id="shadow21-90"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible">
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st2"
+ id="path171"
+ inkscape:connector-curvature="0"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2)" />
+ </g>
+ <path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st6"
+ id="path173"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" />
+ <text
+ x="12.39"
+ y="594.5"
+ class="st4"
+ v:langID="1033"
+ id="text175"
+ style="fill:#feffff;font-family:Calibri"><v:paragraph
+ v:horizAlign="1" /><v:tabList />W 1</text>
+
+ </g>
+ <g
+ id="shape28-95"
+ v:mID="28"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,345.12321,-305.07614)">
+ <title
+ id="title178">Dynamic connector.28</title>
+ <path
+ d="m 0,603 50.38,0"
+ class="st7"
+ id="path180"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape29-100"
+ v:mID="29"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,345.12321,-312.06416)">
+ <title
+ id="title183">Dynamic connector.29</title>
+ <path
+ d="m 0,612 28.33,0 0,-68 22.05,0"
+ class="st7"
+ id="path185"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape30-105"
+ v:mID="30"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,345.12321,-312.06416)">
+ <title
+ id="title188">Dynamic connector.30</title>
+ <path
+ d="m 0,612 28.33,0 0,68 22.05,0"
+ class="st7"
+ id="path190"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape31-110"
+ v:mID="31"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,421.53797,-259.2658)">
+ <title
+ id="title193">Dynamic connector.31</title>
+ <path
+ d="m 0,612 24.42,0 0,-68 25.96,0"
+ class="st7"
+ id="path195"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape32-115"
+ v:mID="32"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,421.53797,-305.07614)">
+ <title
+ id="title198">Dynamic connector.32</title>
+ <path
+ d="m 0,603 50.38,0"
+ class="st7"
+ id="path200"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape33-120"
+ v:mID="33"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,421.53797,-364.86253)">
+ <title
+ id="title203">Dynamic connector.33</title>
+ <path
+ d="m 0,612 24.42,0 0,68 25.96,0"
+ class="st7"
+ id="path205"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <g
+ id="shape34-125"
+ v:mID="34"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,252.79052,-364.86253)">
+ <title
+ id="title208">Dynamic connector.34</title>
+ <path
+ d="m 0,612 26.88,0 0,68 23.5,0"
+ class="st7"
+ id="path210"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" />
+ </g>
+ <text
+ xml:space="preserve"
+ style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;font-family:Sans"
+ x="153.38116"
+ y="165.90149"
+ id="text3106"
+ sodipodi:linespacing="125%"><tspan
+ sodipodi:role="line"
+ x="153.38116"
+ y="165.90149"
+ id="tspan3110"
+ style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Atomic #1</tspan></text>
+<text
+ xml:space="preserve"
+ style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans"
+ x="322.12939"
+ y="165.90149"
+ id="text3106-1"
+ sodipodi:linespacing="125%"><tspan
+ sodipodi:role="line"
+ x="322.12939"
+ y="165.90149"
+ id="tspan3110-4"
+ style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Atomic #2</tspan></text>
+<text
+ xml:space="preserve"
+ style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans"
+ x="491.82089"
+ y="172.79289"
+ id="text3106-0"
+ sodipodi:linespacing="125%"><tspan
+ sodipodi:role="line"
+ x="491.82089"
+ y="172.79289"
+ style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans"
+ id="tspan3923" /></text>
+<text
+ xml:space="preserve"
+ style="font-size:24.84628868px;font-style:normal;font-weight:normal;line-height:125%;letter-spacing:0px;word-spacing:0px;fill:#000000;fill-opacity:1;stroke:none;overflow:visible;font-family:Sans"
+ x="491.02899"
+ y="165.03951"
+ id="text3106-8-5"
+ sodipodi:linespacing="125%"><tspan
+ sodipodi:role="line"
+ x="491.02899"
+ y="165.03951"
+ id="tspan3110-2-1"
+ style="font-size:8.69620132px;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;text-align:center;line-height:125%;writing-mode:lr-tb;text-anchor:middle;fill:#ffffff;fill-opacity:1;font-family:Sans;-inkscape-font-specification:Sans">Single Link</tspan></text>
+<g
+ style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible"
+ id="shape5-22-1"
+ v:mID="5"
+ v:groupContext="shape"
+ transform="matrix(0.77644652,0,0,0.77644652,556.00223,-296.89447)"><title
+ id="title64-5">Circle</title><desc
+ id="desc66-2">RX</desc><v:userDefs><v:ud
+ v:nameU="visVersion"
+ v:val="VT0(15):26" /></v:userDefs><v:textBlock
+ v:margins="rect(4,4,4,4)" /><v:textRect
+ cx="20.5"
+ cy="591.5"
+ width="35.88"
+ height="30.75" /><g
+ id="shadow5-23-7"
+ v:groupContext="shadow"
+ v:shadowOffsetX="0.345598"
+ v:shadowOffsetY="-1.97279"
+ v:shadowType="1"
+ transform="translate(0.345598,1.97279)"
+ class="st1"
+ style="visibility:visible"><path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st2"
+ id="path69-6"
+ inkscape:connector-curvature="0"
+ style="fill:#5b9bd5;fill-opacity:0.22000002;stroke:#5b9bd5;stroke-opacity:0.22000002;filter:url(#filter_2-7)" /></g><path
+ d="m 0,591.5 a 20.5,20.5 0 0 1 41,0 20.5,20.5 0 1 1 -41,0 z"
+ class="st6"
+ id="path71-1"
+ inkscape:connector-curvature="0"
+ style="fill:#ffd965;stroke:#c7c8c8;stroke-width:0.25" /><text
+ x="11.06866"
+ y="596.56067"
+ class="st4"
+ v:langID="1033"
+ id="text73-4"
+ style="fill:#feffff;font-family:Calibri"> TX</text>
+</g><g
+ style="font-size:12px;fill:none;stroke-linecap:square;stroke-miterlimit:3;overflow:visible"
+ id="shape28-95-5"
+ v:mID="28"
+ v:groupContext="shape"
+ v:layerMember="0"
+ transform="matrix(0.77644652,0,0,0.77644652,512.00213,-305.42637)"><title
+ id="title178-7">Dynamic connector.28</title><path
+ d="m 0,603 50.38,0"
+ class="st7"
+ id="path180-6"
+ inkscape:connector-curvature="0"
+ style="stroke:#5b9bd5;stroke-width:1;stroke-linecap:round;stroke-linejoin:round;marker-end:url(#mrkr4-33)" /></g></g>
+</svg>
diff --git a/doc/guides/prog_guide/index.rst b/doc/guides/prog_guide/index.rst
index 7f825cb..2d23b82 100644
--- a/doc/guides/prog_guide/index.rst
+++ b/doc/guides/prog_guide/index.rst
@@ -57,6 +57,7 @@ Programmer's Guide
multi_proc_support
kernel_nic_interface
thread_safety_dpdk_functions
+ eventdev
qos_framework
power_man
packet_classif_access_ctrl
--
2.7.4
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide
2017-03-15 17:21 [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide Harry van Haaren
@ 2017-03-30 9:14 ` Burakov, Anatoly
2017-03-30 13:36 ` Jerin Jacob
1 sibling, 0 replies; 6+ messages in thread
From: Burakov, Anatoly @ 2017-03-30 9:14 UTC (permalink / raw)
To: Van Haaren, Harry, dev; +Cc: Van Haaren, Harry
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Harry van Haaren
> Sent: Wednesday, March 15, 2017 5:21 PM
> To: dev@dpdk.org
> Cc: Van Haaren, Harry <harry.van.haaren@intel.com>
> Subject: [dpdk-dev] [PATCH] doc: add eventdev library to programmers
> guide
>
> This commit adds an entry in the programmers guide explaining the
> eventdev library.
>
> The rte_event struct, queues and ports are explained.
> An API walktrough of a simple two stage atomic pipeline provides the reader
> with a step by step overview of the expected usage of the Eventdev API.
>
> Signed-off-by: Harry van Haaren <harry.van.haaren@intel.com>
>
> ---
Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide
2017-03-15 17:21 [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide Harry van Haaren
2017-03-30 9:14 ` Burakov, Anatoly
@ 2017-03-30 13:36 ` Jerin Jacob
2017-03-30 15:59 ` Van Haaren, Harry
2017-03-31 12:43 ` Thomas Monjalon
1 sibling, 2 replies; 6+ messages in thread
From: Jerin Jacob @ 2017-03-30 13:36 UTC (permalink / raw)
To: Harry van Haaren; +Cc: dev, thomas.monjalon, hemant.agrawal, bruce.richardson
On Wed, Mar 15, 2017 at 05:21:18PM +0000, Harry van Haaren wrote:
> This commit adds an entry in the programmers guide
> explaining the eventdev library.
>
> The rte_event struct, queues and ports are explained.
> An API walktrough of a simple two stage atomic pipeline
> provides the reader with a step by step overview of the
> expected usage of the Eventdev API.
Thanks Harry for writing this document.
Overall it looks good.
I have a few thoughts on this document and next steps of eventdev:
1) Need more use cases and associated example applications
The initial version talks only about queue based event pipelining,
I think, we need to add another use case like following in the eventdev
programmers guide with associated example applications.
a) automatic multicore scaling
b) dynamic load balancing,
c) flow based pipelining
d) packet ingress order maintenance
e) synchronization services.
2) ethdev integration is pending.
There are some ethdev(NIC) HW features associated with HW eventdev
implementations. For example, The given example in the programming
guide has dedicated Rx and Tx cores, In HW ethdev and HW eventdev case,
- NIC HW can produce and inject the events to eventdev. No need for
additional Rx core push the data to eventdev on those HW
- In Some NIC HW without SW lock, multiple cores can access the same tx queue
hence the need for additional TX core may not valid on those HW.
IMO, Addressing all the use case with generic example application and
ethdev integration with eventdev would call for minor changes in the
eventdev API and the programmer guide content.
Considering the above points, I would like to share the current
eventdev status and propose the next step for eventdev.
Current eventdev status:
1) Eventdev specification more or less complete
2) Two eventdev driver implementations are ready(completed the review) along
with unit test cases.
a) An eventdev SW driver from Intel
b) An eventdev HW driver from Cavium
Pending work:
1) More generic example applications with additional eventdev uses case
backed by programming guide.
IMO, programming guide will be complete only with generic example
applications.IMHO, we can add the programming guide once we have generic examples
2) Minor change in eventdev specification to accommodate ethdev integration
and example generic application with ethdev and eventdev for packet
forwarding.
If noone is working this, I am planning to work on item 1 and 2 to
complete eventdev work.
Based on the above status, I would like to merge eventdev to DPDK
v17.05 with EXPERIMENTAL tag and address pending work in v17.08 and
remove the EXPERIMENTAL tag.
Thomas and all,
Thoughts ?
NOTE:
Keeping the EXPERIMENTAL status to allow minor change in the API without
ABI issue when we address the pending work and I guess NXP eventdev also
lined up for v17.08 and they may need a minor change in the API
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide
2017-03-30 13:36 ` Jerin Jacob
@ 2017-03-30 15:59 ` Van Haaren, Harry
2017-03-31 12:43 ` Thomas Monjalon
1 sibling, 0 replies; 6+ messages in thread
From: Van Haaren, Harry @ 2017-03-30 15:59 UTC (permalink / raw)
To: Jerin Jacob; +Cc: dev, thomas.monjalon, hemant.agrawal, Richardson, Bruce
> From: Jerin Jacob [mailto:jerin.jacob@caviumnetworks.com]
> Sent: Thursday, March 30, 2017 2:37 PM
> To: Van Haaren, Harry <harry.van.haaren@intel.com>
> Cc: dev@dpdk.org; thomas.monjalon@6wind.com; hemant.agrawal@nxp.com; Richardson, Bruce
> <bruce.richardson@intel.com>
> Subject: Re: [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide
>
> On Wed, Mar 15, 2017 at 05:21:18PM +0000, Harry van Haaren wrote:
> > This commit adds an entry in the programmers guide
> > explaining the eventdev library.
> >
> > The rte_event struct, queues and ports are explained.
> > An API walktrough of a simple two stage atomic pipeline
> > provides the reader with a step by step overview of the
> > expected usage of the Eventdev API.
>
>
> Thanks Harry for writing this document.
> Overall it looks good.
Thanks for the thanks!
> I have a few thoughts on this document and next steps of eventdev:
>
> 1) Need more use cases and associated example applications
>
> The initial version talks only about queue based event pipelining,
> I think, we need to add another use case like following in the eventdev
> programmers guide with associated example applications.
> a) automatic multicore scaling
> b) dynamic load balancing,
> c) flow based pipelining
> d) packet ingress order maintenance
> e) synchronization services.
Agreed some sample apps would ease starting to use the eventdev API.
> 2) ethdev integration is pending.
>
> There are some ethdev(NIC) HW features associated with HW eventdev
> implementations. For example, The given example in the programming
> guide has dedicated Rx and Tx cores, In HW ethdev and HW eventdev case,
>
> - NIC HW can produce and inject the events to eventdev. No need for
> additional Rx core push the data to eventdev on those HW
> - In Some NIC HW without SW lock, multiple cores can access the same tx queue
> hence the need for additional TX core may not valid on those HW.
Right - these details will need to be worked out as the PMDs are developed.
> IMO, Addressing all the use case with generic example application and
> ethdev integration with eventdev would call for minor changes in the
> eventdev API and the programmer guide content.
Yes a generic sample app would be great, I'll post the sample app that we've been using for testing the SW PMD to patchwork in the next weeks if possible. It is a multi-stage pipeline based application, similar to that described in the prog-guide documentation.
> Considering the above points, I would like to share the current
> eventdev status and propose the next step for eventdev.
>
> Current eventdev status:
> 1) Eventdev specification more or less complete
> 2) Two eventdev driver implementations are ready(completed the review) along
> with unit test cases.
> a) An eventdev SW driver from Intel
> b) An eventdev HW driver from Cavium
>
> Pending work:
> 1) More generic example applications with additional eventdev uses case
> backed by programming guide.
> IMO, programming guide will be complete only with generic example
> applications.IMHO, we can add the programming guide once we have generic examples
> 2) Minor change in eventdev specification to accommodate ethdev integration
> and example generic application with ethdev and eventdev for packet
> forwarding.
>
> If noone is working this, I am planning to work on item 1 and 2 to
> complete eventdev work.
Generic sample apps that work with all PMDs would be great, as above I'll share the example pipeline app asap.
Regarding API changes, yes we can deal with them as they arise during implementation.
> Based on the above status, I would like to merge eventdev to DPDK
> v17.05 with EXPERIMENTAL tag and address pending work in v17.08 and
> remove the EXPERIMENTAL tag.
>
> Thomas and all,
>
> Thoughts ?
>
> NOTE:
>
> Keeping the EXPERIMENTAL status to allow minor change in the API without
> ABI issue when we address the pending work and I guess NXP eventdev also
> lined up for v17.08 and they may need a minor change in the API
This seems a good suggestion to me, as it enables us to move quickly in terms of API/ABI (if required for new/updated PMDs) but still ensures that the eventdev becomes available to DPDK mainline users too.
No objections to this approach here, -Harry
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide
2017-03-30 13:36 ` Jerin Jacob
2017-03-30 15:59 ` Van Haaren, Harry
@ 2017-03-31 12:43 ` Thomas Monjalon
2017-03-31 13:31 ` Jerin Jacob
1 sibling, 1 reply; 6+ messages in thread
From: Thomas Monjalon @ 2017-03-31 12:43 UTC (permalink / raw)
To: Jerin Jacob; +Cc: Harry van Haaren, dev, hemant.agrawal, bruce.richardson
2017-03-30 19:06, Jerin Jacob:
> 2) ethdev integration is pending.
What is pending exactly?
[...]
> Based on the above status, I would like to merge eventdev to DPDK
> v17.05 with EXPERIMENTAL tag and address pending work in v17.08 and
> remove the EXPERIMENTAL tag.
>
> Thomas and all,
>
> Thoughts ?
>
> NOTE:
>
> Keeping the EXPERIMENTAL status to allow minor change in the API without
> ABI issue when we address the pending work and I guess NXP eventdev also
> lined up for v17.08 and they may need a minor change in the API
Yes, generally speaking, it is a good idea to put the experimental tag
for the first release of an API.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide
2017-03-31 12:43 ` Thomas Monjalon
@ 2017-03-31 13:31 ` Jerin Jacob
0 siblings, 0 replies; 6+ messages in thread
From: Jerin Jacob @ 2017-03-31 13:31 UTC (permalink / raw)
To: Thomas Monjalon; +Cc: Harry van Haaren, dev, hemant.agrawal, bruce.richardson
On Fri, Mar 31, 2017 at 02:43:15PM +0200, Thomas Monjalon wrote:
> 2017-03-30 19:06, Jerin Jacob:
> > 2) ethdev integration is pending.
>
> What is pending exactly?
As a standalone library, Nothing is pending in eventdev.
There are some HW features in the NIC side in some Hardwares that helps
in offloading eventdev related work in NIC. For example,
In the RX side, Some actor is required to inject the events/packets to
eventdev. Some Hardware is capable of offloading that work in NIC.
if that HW feature is not available then core can do the same.
On the TX side, N cores can send the packets to the same TX queue
WITHOUT SW lock. This helps in proper scaling, If that HW feature is not available
in NIC side then core can decide to bind only one tx queue per core.
On the RX side, I don't expect any change in ethdev API.
On the TX side, We may need to have some capability to expose the lock-less
features(I haven't thought about the abstraction details fully).
What I meant by pending here is that, Some future work is required to
enable, abstracting those HW difference and to have portable example
eventdev applications.
>
> [...]
> > Based on the above status, I would like to merge eventdev to DPDK
> > v17.05 with EXPERIMENTAL tag and address pending work in v17.08 and
> > remove the EXPERIMENTAL tag.
> >
> > Thomas and all,
> >
> > Thoughts ?
> >
> > NOTE:
> >
> > Keeping the EXPERIMENTAL status to allow minor change in the API without
> > ABI issue when we address the pending work and I guess NXP eventdev also
> > lined up for v17.08 and they may need a minor change in the API
>
> Yes, generally speaking, it is a good idea to put the experimental tag
> for the first release of an API.
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2017-03-31 13:31 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-15 17:21 [dpdk-dev] [PATCH] doc: add eventdev library to programmers guide Harry van Haaren
2017-03-30 9:14 ` Burakov, Anatoly
2017-03-30 13:36 ` Jerin Jacob
2017-03-30 15:59 ` Van Haaren, Harry
2017-03-31 12:43 ` Thomas Monjalon
2017-03-31 13:31 ` Jerin Jacob
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).