* [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem
@ 2020-03-18 21:35 jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 01/26] graph: define the public API for graph support jerinj
` (26 more replies)
0 siblings, 27 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark,
pbhagavatula, ndabilpuram, Jerin Jacob
From: Jerin Jacob <jerinj@marvell.com>
It is the v1 version of the DPDK graph support based on the following
RFC http://patches.dpdk.org/cover/65432/
This patch set contains an implementation of graph architecture for
packet processing using DPDK primitives.
Using graph traversal for packet processing is a proven architecture
that has been implemented in various open source libraries.
Graph architecture for packet processing enables abstracting the data
processing functions as “nodes” and “links” them together to create a
complex “graph” to create reusable/modular data processing functions.
The patchset further includes performance enhancements and modularity
to the DPDK as discussed in more detail below.
RFC..v1:
--------
1) Split the patch to more logical ones for review.
2) Added doxygen comments for the API
3) Code cleanup
4) Additional performance improvements.
Delta between l3fwd and l3fwd-graph is negligible now.
(~1%) on octeontx2.
5) Added SIMD routines for x86 in additional to arm64.
Pending items (Will be addressed in v2)
-------------------------------------------------
1) Add documentation as a patch for programming guide and l3fwd-graph
user guide.
Addional nodes planned for v20.08
----------------------------------
1) Packet classification node
2) Support for IPV6 LPM node
This patchset contains
-----------------------------
1) The API definition to "create" nodes and "link" together to create a
"graph" for packet processing. See, lib/librte_graph/rte_graph.h
2) The Fast path API definition for the graph walker and enqueue
function used by the workers. See, lib/librte_graph/rte_graph_worker.h
3) Optimized SW implementation for (1) and (2). See, lib/librte_graph/
4) Test case to verify the graph infrastructure functionality
See, app/test/test_graph.c
5) Performance test cases to evaluate the cost of graph walker and nodes
enqueue fast-path function for various combinations.
See app/test/test_graph_perf.c
6) Packet processing nodes(Null, Rx, Tx, Pkt drop, IPV4 rewrite, IPv4
lookup)
using graph infrastructure. See lib/librte_node/*
7) An example application to showcase l3fwd
(functionality same as existing examples/l3fwd) using graph
infrastructure and use packets processing nodes (item (6)). See examples/l3fwd-graph/.
Performance
-----------
1) Graph walk and node enqueue overhead can be tested with performance
test case application [1]
# If all packets go from a node to another node (we call it as
# "homerun") then it will be just a pointer swap for a burst of packets.
# In the worst case, a couple of handful cycles to move an object from a
node to another node.
2) Performance comparison with existing l3fwd (The complete static code
with out any nodes) vs modular l3fwd-graph with 5 nodes
(ip4_lookup, ip4_rewrite, ethdev_tx, ethdev_rx, pkt_drop).
Here is graphical representation of the l3fwd-graph as Graphviz dot
file:
http://bit.ly/39UPPGm
# l3fwd-graph performance is -1.2% wrt static l3fwd.
# We have simulated the similar test with existing librte_pipeline
# application [4].
ip_pipline application is -48.62% wrt static l3fwd.
The above results are on octeontx2. It may vary on other platforms.
The platforms with higher L1 and L2 caches will have further better
performance.
Tested architectures:
--------------------
1) AArch64
2) X86
Graph library Features
----------------------
1) Nodes as plugins
2) Support for out of tree nodes
3) Multi-process support.
4) Low overhead graph walk and node enqueue
5) Low overhead statistics collection infrastructure
6) Support to export the graph as a Graphviz dot file.
See rte_graph_export()
Example of exported graph: http://bit.ly/2PqbqOy
7) Allow having another graph walk implementation
in the future by segregating the fast path and slow path code.
Advantages of Graph architecture:
---------------------------------
1) Memory latency is the enemy for high-speed packet processing,
moving the similar packet processing code to a node will reduce
the I cache and D caches misses.
2) Exploits the probability that most packets will follow the same nodes
in the graph.
3) Allow SIMD instructions for packet processing of the node.
4) The modular scheme allows having reusable nodes for the consumers.
5) The modular scheme allows us to abstract the vendor HW specific
optimizations as a node.
Why Graph architecture
-----------------------
1) We believe, Graph architecture provides the best performance for
reusable/modular packet processing framework.
Since DPDK does not have it, it is good to have it in DPDK.
2) Based on our experience, NPU HW accelerates are so different than one
vendor
to another vendor. Going forward, We believe, API abstraction may not be
enough
abstract the difference in HW. The Vendor-specific nodes can abstract
the HW
differences and reuse generic the nodes as needed.
This would help both the silicon vendors and DPDK end users.
3) The framework enables the protocol stack as use native mbuf for
graph processing to avoid any conversion between the formats for
better performance.
4) DPDK becomes the "goto library" for userspace HW acceleration.
It is good to have native Graph packet processing library in DPDK.
5) Obviously, Our customers are interested in Graph library in DPDK :-)
Identified tweaking for better performance on different targets
---------------------------------------------------------------
1) Test with various burst size values (256, 128, 64, 32) using
CONFIG_RTE_GRAPH_BURST_SIZE config option.
Based on our testing, on x86 and arm64 servers, The sweet spot is 256
burst size.
While on arm64 embedded SoCs, it is either 64 or 128.
2) Disable node statistics (use CONFIG_RTE_LIBRTE_GRAPH_STATS config
option)
if not needed.
3) Use arm64 optimized memory copy for arm64 architecture by
selecting CONFIG_RTE_ARCH_ARM64_MEMCPY.
Commands to run tests
---------------------
[1]
perf test:
echo "graph_perf_autotest" | sudo ./build/app/test/dpdk-test -c 0x30
[2]
functionality test:
echo "graph_autotest" | sudo ./build/app/test/dpdk-test -c 0x30
[3]
l3fwd-graph:
./l3fwd-graph -c 0x100 -- -p 0x3 --config="(0, 0, 8)" -P
[4]
# ./ip_pipeline --c 0xff0000 -- -s route.cli
Route.cli: (Copy paste to the shell to avoid dos format issues)
https://pastebin.com/raw/B4Ktx7TT
Programming guide and API walk-through
--------------------------------------
# Anatomy of Node:
~~~~~~~~~~~~~~~~~
See the
https://github.com/jerinjacobk/share/blob/master/Anatomy_of_a_node.svg
The above diagram depicts the anatomy of a node.
The node is the basic building block of the graph framework.
A node consists of:
a) process():
The callback function will be invoked by worker thread using
rte_graph_walk() function when there is data to be processed by the
node.
A graph node process the function using process() and enqueue to next
downstream node using rte_node_enqueue*() function.
b) Context memory:
It is memory allocated by the library to store the node-specific context
information. which will be used by process(), init(), fini() callbacks.
c) init():
The callback function which will be invoked by rte_graph_create() on
when a node
gets attached to a graph.
d) fini():
The callback function which will be invoked by rte_graph_destroy() on
when a node
gets detached to a graph.
e) Node name:
It is the name of the node. When a node registers to graph library, the
library
gives the ID as rte_node_t type. Both ID or Name shall be used lookup
the node.
rte_node_from_name(), rte_node_id_to_name() are the node lookup
functions.
f) nb_edges:
Number of downstream nodes connected to this node. The next_nodes[]
stores the
downstream nodes objects. rte_node_edge_update() and
rte_node_edge_shrink()
functions shall be used to update the next_node[] objects. Consumers of
the node
APIs are free to update the next_node[] objects till rte_graph_create()
invoked.
g) next_node[]:
The dynamic array to store the downstream nodes connected to this node.
# Node creation and registration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a) Node implementer creates the node by implementing ops and attributes
of
'struct rte_node_register'
b) The library registers the node by invoking RTE_NODE_REGISTER on
library load
using the constructor scheme.
The constructor scheme used here to support multi-process.
# Link the Nodes to create the graph topology
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
See the
https://github.com/jerinjacobk/share/blob/master/Link_the_nodes.svg
The above diagram shows a graph topology after linking the N nodes.
Once nodes are available to the program, Application or node public API
functions
can links them together to create a complex packet processing graph.
There are multiple different types of strategies to link the nodes.
Method a) Provide the next_nodes[] at the node registration time.
See 'struct rte_node_register::nb_edges'. This is a use case to address
the static
node scheme where one knows upfront the next_nodes[] of the node.
Method b) Use rte_node_edge_get(), rte_node_edge_update(),
rte_node_edge_shrink() to
Update the next_nodes[] links for the node dynamically.
Method c) Use rte_node_clone() to clone a already existing node.
When rte_node_clone() invoked, The library, would clone all the
attributes
of the node and creates a new one. The name for cloned node shall be
"parent_node_name-user_provided_name". This method enables the use case
of Rx and Tx
nodes where multiple of those nodes need to be cloned based on the
number of CPU
available in the system. The cloned nodes will be identical, except the
"context memory".
Context memory will have information of port, queue pair incase of Rx
and Tx ethdev nodes.
# Create the graph object
~~~~~~~~~~~~~~~~~~~~~~~~~
Now that the nodes are linked, Its time to create a graph by including
the required nodes. The application can provide a set of node patterns
to
form a graph object.
The fnmatch() API used underneath for the pattern matching to include
the required nodes.
The rte_graph_create() API shall be used to create the graph.
Example of a graph object creation:
{"ethdev_rx_0_0", ipv4-*, ethdev_tx_0_*"}
In the above example, A graph object will be created with ethdev Rx
node of port 0 and queue 0, all ipv4* nodes in the system,
and ethdev tx node of port 0 with all queues.
# Multi core graph processing
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the current graph library implementation, specifically,
rte_graph_walk() and rte_node_enqueue* fast path API functions
are designed to work on single-core to have better performance.
The fast path API works on graph object, So the multi-core graph
processing strategy would be to create graph object PER WORKER.
# In fast path:
~~~~~~~~~~~~~~~
Typical fast-path code looks like below, where the application
gets the fast-path graph object through rte_graph_lookup()
on the worker thread and run the rte_graph_walk() in a tight loop.
struct rte_graph *graph = rte_graph_lookup("worker0");
while (!done) {
rte_graph_walk(graph);
}
# Context update when graph walk in action
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The fast-path object for the node is `struct rte_node`.
It may be possible that in slow-path or after the graph walk-in action,
the user needs to update the context of the node hence access to
struct rte_node * memory.
rte_graph_foreach_node(), rte_graph_node_get(),
rte_graph_node_get_by_name()
APIs can be used to to get the struct rte_node*.
rte_graph_foreach_node() iterator
function works on struct rte_graph * fast-path graph object while others
works on graph ID or name.
# Get the node statistics using graph cluster
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The user may need to know the aggregate stats of the node across
multiple graph objects. Especially the situation where each
graph object bound to a worker thread.
Introduced a graph cluster object for statistics.
rte_graph_cluster_stats_create()
shall be used for creating a graph cluster with multiple graph objects
and
rte_graph_cluster_stats_get() to get the aggregate node statistics.
An example statistics output from rte_graph_cluster_stats_get()
+-----------+------------+-------------+---------------+------------+---------------+-----------+
|Node |calls |objs |realloc_count |objs/call
|objs/sec(10E6) |cycles/call|
+------------------------+-------------+---------------+------------+---------------+-----------+
|node0 |12977424 |3322220544 |5 |256.000
|3047.151872 |20.0000 |
|node1 |12977653 |3322279168 |0 |256.000
|3047.210496 |17.0000 |
|node2 |12977696 |3322290176 |0 |256.000
|3047.221504 |17.0000 |
|node3 |12977734 |3322299904 |0 |256.000
|3047.231232 |17.0000 |
|node4 |12977784 |3322312704 |1 |256.000
|3047.243776 |17.0000 |
|node5 |12977825 |3322323200 |0 |256.000
|3047.254528 |17.0000 |
+-----------+------------+-------------+---------------+------------+---------------+-----------+
# Node writing guide lines
~~~~~~~~~~~~~~~~~~~~~~~~~~
The process() function of a node is fast-path function and that needs to
be written
carefully to achieve max performance.
Broadly speaking, there are two different types of nodes.
1) First kind of nodes are those that have a fixed next_nodes[] for the
complete burst (like ethdev_rx, ethdev_tx) and it is simple to write.
Process() function can move the obj burst to the next node either using
rte_node_next_stream_move() or using rte_node_next_stream_get() and
rte_node_next_stream_put().
2) The second kind of such node is `intermediate nodes` that decide what
is the next_node[]
to send to on a per-packet basis. In these nodes,
a) Firstly, there has to be the best possible packet processing logic.
b) Secondly, each packet needs to be queued to its next node.
This can be done using rte_node_enqueue_[x1|x2|x4]() api's if they are
to single next or
rte_node_enqueue_next() that takes array of nexts.
In scenario where multiple intermediate nodes are present but most of
the time each node using same next node for all its packets, cost of moving every
pointer from current node's stream to next node's stream could be avoided.
This is called home run and rte_node_next_stream_move() could be used to
just move stream from current node to next node with least number of
cycles.
Since this can be avoided only in the case where all the packets are
destined to the same next node, node implementation should be also having worst
case handling where every packet could be going to different next node.
Example of intermediate node implementation with home run:
a) Start with speculation that next_node = ctx->next_node.
This could be the next_node application used in the previous function
call of this node.
b) Get the next_node stream array with required space using
rte_node_next_stream_get(next_node, space)
c) while n_left_from > 0 // Pkts left to be sent
prefetch next pkt_set and process current pkt_set to find their next node
d) if all the next nodes of the current pkt_set match speculated next
node, just count them as successfully speculated('last_spec') till now
and continue the loop without actually moving them to the next node.
else if there is a mismatch,
copy all the pkt_set pointers that were 'last_spec' and move
the current pkt_set to their respective next's nodes using
rte_enqueue_next_x1(). Also one of the next_node can be updated
as
speculated next_node if it is more probable.
Finally reset 'last_spec' to zero.
e) if n_left_from != 0
goto c) to process remaining pkts.
f) if last_spec == nb_objs,
All the objects passed were successfully speculated to single next
node.
So, the current stream can be moved to next node using
rte_node_next_stream_move(node, next_node). This is home run
where memcpy of buffer pointers to next node is avoided.
g) Update the ctx->next_node with more probable next node.
# In-tree node documentation
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a) librte_node/ethdev_rx.c:
This node does rte_eth_rx_burst() into stream buffer passed to it
(src node stream) and does rte_node_next_stream_move() only when
there are packets received. Each rte_node works on only on
one rx port and queue that it gets from node->context.
For each (port X, rx_queue Y), a rte_node is cloned from
ethdev_rx_base_node as "ethdev_rx-X-Y" in rte_node_eth_config()
along with updating node->context. Each graph needs to be associated
with a unique rte_node for a (port, rx_queue).
b) librte_node/ethdev_tx.c:
This node does rte_eth_tx_burst() for a burst of objs received by
it.
It sends the burst to a fixed Tx Port and Queue information from
node->context. For each (port X), this rte_node is cloned from
ethdev_tx_node_base as "ethdev_tx-X" in rte_node_eth_config()
along with updating node->context.
Since each graph doesn't need more than one Txq, per port,
a Txq is assigned based on graph id to each rte_node instance.
Each graph needs to be associated with a rte_node for each
(port).
c) librte_node/pkt_drop.c:
This node frees all the objects passed to it considering them as
rte_mbufs that need to be freed.
d) librte_node/ip4_lookup.c:
This node is an intermediate node that does lpm lookup for the
receive ipv4 packets and the result determines each packets next node.
a) On successful lpm lookup, the result contains the nex_node id
and next-hop id with which the packet needs to be further
processed.
b) On lpm lookup failure, objects are redirected to pkt_drop node.
rte_node_ip4_route_add() is control path API to add ipv4 routes.
To achieve home run, we use rte_node_stream_move() as mentioned in
above sections.
e) librte_node/ip4_rewrite.c:
This node gets packets from ip4_lookup node with next-hop id for
each packet is embedded in rte_node_mbuf_priv1(mbuf)->nh. This id is
used to determine the L2 header to be written to the pkt before sending
the pkt out to a particular ethdev_tx node.
rte_node_ip4_rewrite_add() is control path API to add next-hop info.
f) librte_node/null.c:
This is null node that just ignores the set of objects passed to
it and reports that all are processed.
Jerin Jacob (12):
graph: define the public API for graph support
graph: implement node registration
graph: implement node operations
graph: implement node debug routines
graph: implement internal graph operation helpers
graph: populate fastpath memory for graph reel
graph: implement create and destroy APIs
graph: implement graph operation APIs
graph: implement Graphviz export
graph: implement debug routines
graph: implement stats support
graph: implement fastpath API routines
Kiran Kumar K (2):
graph: add unit test case
node: add ipv4 rewrite node
Nithin Dabilpuram (10):
node: add log infra and null node
node: add ethdev Rx node
node: add ethdev Tx node
node: add ethdev Rx and Tx node ctrl API
node: ipv4 lookup for arm64
node: add ipv4 rewrite and lookup ctrl API
node: add pkt drop node
l3fwd-graph: add graph based l3fwd skeleton
l3fwd-graph: add ethdev configuration changes
l3fwd-graph: add graph config and main loop
Pavan Nikhilesh (2):
graph: add performance testcase
node: ipv4 lookup for x86
MAINTAINERS | 13 +
app/test/Makefile | 5 +
app/test/meson.build | 11 +-
app/test/test_graph.c | 820 +++++++++++++++++
app/test/test_graph_perf.c | 1057 ++++++++++++++++++++++
config/common_base | 12 +
config/rte_config.h | 4 +
doc/api/doxy-api-index.md | 5 +
doc/api/doxy-api.conf.in | 2 +
examples/Makefile | 3 +
examples/l3fwd-graph/Makefile | 58 ++
examples/l3fwd-graph/main.c | 1113 ++++++++++++++++++++++++
examples/l3fwd-graph/meson.build | 13 +
examples/meson.build | 6 +-
lib/Makefile | 6 +
lib/librte_graph/Makefile | 28 +
lib/librte_graph/graph.c | 590 +++++++++++++
lib/librte_graph/graph_debug.c | 84 ++
lib/librte_graph/graph_ops.c | 169 ++++
lib/librte_graph/graph_populate.c | 234 +++++
lib/librte_graph/graph_private.h | 346 ++++++++
lib/librte_graph/graph_stats.c | 406 +++++++++
lib/librte_graph/meson.build | 11 +
lib/librte_graph/node.c | 421 +++++++++
lib/librte_graph/rte_graph.h | 786 +++++++++++++++++
lib/librte_graph/rte_graph_version.map | 47 +
lib/librte_graph/rte_graph_worker.h | 541 ++++++++++++
lib/librte_node/Makefile | 30 +
lib/librte_node/ethdev_ctrl.c | 115 +++
lib/librte_node/ethdev_rx.c | 221 +++++
lib/librte_node/ethdev_rx_priv.h | 81 ++
lib/librte_node/ethdev_tx.c | 86 ++
lib/librte_node/ethdev_tx_priv.h | 62 ++
lib/librte_node/ip4_lookup.c | 631 ++++++++++++++
lib/librte_node/ip4_rewrite.c | 325 +++++++
lib/librte_node/ip4_rewrite_priv.h | 77 ++
lib/librte_node/log.c | 14 +
lib/librte_node/meson.build | 8 +
lib/librte_node/node_private.h | 96 ++
lib/librte_node/null.c | 23 +
lib/librte_node/pkt_drop.c | 26 +
lib/librte_node/rte_node_eth_api.h | 70 ++
lib/librte_node/rte_node_ip4_api.h | 87 ++
lib/librte_node/rte_node_version.map | 9 +
lib/meson.build | 5 +-
meson.build | 1 +
mk/rte.app.mk | 2 +
47 files changed, 8755 insertions(+), 5 deletions(-)
create mode 100644 app/test/test_graph.c
create mode 100644 app/test/test_graph_perf.c
create mode 100644 examples/l3fwd-graph/Makefile
create mode 100644 examples/l3fwd-graph/main.c
create mode 100644 examples/l3fwd-graph/meson.build
create mode 100644 lib/librte_graph/Makefile
create mode 100644 lib/librte_graph/graph.c
create mode 100644 lib/librte_graph/graph_debug.c
create mode 100644 lib/librte_graph/graph_ops.c
create mode 100644 lib/librte_graph/graph_populate.c
create mode 100644 lib/librte_graph/graph_private.h
create mode 100644 lib/librte_graph/graph_stats.c
create mode 100644 lib/librte_graph/meson.build
create mode 100644 lib/librte_graph/node.c
create mode 100644 lib/librte_graph/rte_graph.h
create mode 100644 lib/librte_graph/rte_graph_version.map
create mode 100644 lib/librte_graph/rte_graph_worker.h
create mode 100644 lib/librte_node/Makefile
create mode 100644 lib/librte_node/ethdev_ctrl.c
create mode 100644 lib/librte_node/ethdev_rx.c
create mode 100644 lib/librte_node/ethdev_rx_priv.h
create mode 100644 lib/librte_node/ethdev_tx.c
create mode 100644 lib/librte_node/ethdev_tx_priv.h
create mode 100644 lib/librte_node/ip4_lookup.c
create mode 100644 lib/librte_node/ip4_rewrite.c
create mode 100644 lib/librte_node/ip4_rewrite_priv.h
create mode 100644 lib/librte_node/log.c
create mode 100644 lib/librte_node/meson.build
create mode 100644 lib/librte_node/node_private.h
create mode 100644 lib/librte_node/null.c
create mode 100644 lib/librte_node/pkt_drop.c
create mode 100644 lib/librte_node/rte_node_eth_api.h
create mode 100644 lib/librte_node/rte_node_ip4_api.h
create mode 100644 lib/librte_node/rte_node_version.map
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 01/26] graph: define the public API for graph support
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 02/26] graph: implement node registration jerinj
` (25 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Thomas Monjalon, Bruce Richardson, John McNamara,
Marko Kovacevic, Jerin Jacob, Kiran Kumar K
Cc: dev, david.marchand, mdr, mattias.ronnblom, pbhagavatula, ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Graph architecture abstracts the data processing functions as
"node" and "link" them together to create a complex "graph" to enable
reusable/modular data processing functions.
These APIs enables graph framework operations such as create, lookup,
dump and destroy on graph and node operations such as clone,
edge update, and edge shrink, etc. The API also allows creating the
stats cluster to monitor per graph and per node stats.
This patch defines the public API for graph support.
This patch also adds support for the build infrastructure and
update the MAINTAINERS file for the graph subsystem.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
MAINTAINERS | 5 +
config/common_base | 7 +
config/rte_config.h | 4 +
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
lib/Makefile | 3 +
lib/librte_graph/Makefile | 22 +
lib/librte_graph/graph.c | 5 +
lib/librte_graph/meson.build | 11 +
lib/librte_graph/rte_graph.h | 786 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 3 +
lib/meson.build | 2 +-
mk/rte.app.mk | 1 +
13 files changed, 850 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_graph/Makefile
create mode 100644 lib/librte_graph/graph.c
create mode 100644 lib/librte_graph/meson.build
create mode 100644 lib/librte_graph/rte_graph.h
create mode 100644 lib/librte_graph/rte_graph_version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index c3785554f..32d0ea032 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1468,6 +1468,11 @@ F: examples/bpf/
F: app/test/test_bpf.c
F: doc/guides/prog_guide/bpf_lib.rst
+Graph - EXPERIMENTAL
+M: Jerin Jacob <jerinj@marvell.com>
+M: Kiran Kumar K <kirankumark@marvell.com>
+F: lib/librte_graph/
+
Test Applications
-----------------
diff --git a/config/common_base b/config/common_base
index 7ca2f28b1..04a96aef5 100644
--- a/config/common_base
+++ b/config/common_base
@@ -1075,6 +1075,13 @@ CONFIG_RTE_LIBRTE_BPF_ELF=n
#
CONFIG_RTE_LIBRTE_IPSEC=y
+#
+# Compile librte_graph
+#
+CONFIG_RTE_LIBRTE_GRAPH=y
+CONFIG_RTE_GRAPH_BURST_SIZE=256
+CONFIG_RTE_LIBRTE_GRAPH_STATS=y
+
#
# Compile the test application
#
diff --git a/config/rte_config.h b/config/rte_config.h
index d30786bc0..e9201fd46 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -98,6 +98,10 @@
/* KNI defines */
#define RTE_KNI_PREEMPT_DEFAULT 1
+/* rte_graph defines */
+#define RTE_GRAPH_BURST_SIZE 256
+#define RTE_LIBRTE_GRAPH_STATS 1
+
/****** driver defines ********/
/* QuickAssist device */
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index dff496be0..5cc50f750 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -159,6 +159,7 @@ The public API headers are grouped by topics:
* [pipeline] (@ref rte_pipeline.h)
[port_in_action] (@ref rte_port_in_action.h)
[table_action] (@ref rte_table_action.h)
+ * [graph] (@ref rte_graph.h):
- **basic**:
[approx fraction] (@ref rte_approx.h),
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 1c4392eec..759a7213e 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -33,6 +33,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/lib/librte_eventdev \
@TOPDIR@/lib/librte_fib \
@TOPDIR@/lib/librte_flow_classify \
+ @TOPDIR@/lib/librte_graph \
@TOPDIR@/lib/librte_gro \
@TOPDIR@/lib/librte_gso \
@TOPDIR@/lib/librte_hash \
diff --git a/lib/Makefile b/lib/Makefile
index 46b91ae1a..1f572b659 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -119,6 +119,9 @@ DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
DIRS-$(CONFIG_RTE_LIBRTE_RCU) += librte_rcu
DEPDIRS-librte_rcu := librte_eal
+DIRS-$(CONFIG_RTE_LIBRTE_GRAPH) += librte_graph
+DEPDIRS-librte_graph := librte_eal
+
ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
endif
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
new file mode 100644
index 000000000..26fe514f3
--- /dev/null
+++ b/lib/librte_graph/Makefile
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_graph.a
+
+CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_graph_version.map
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+
+# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
new file mode 100644
index 000000000..a55bf443a
--- /dev/null
+++ b/lib/librte_graph/graph.c
@@ -0,0 +1,5 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include "rte_graph.h"
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
new file mode 100644
index 000000000..455cf2ba5
--- /dev/null
+++ b/lib/librte_graph/meson.build
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+name = 'graph'
+
+sources = files('graph.c')
+headers = files('rte_graph.h')
+allow_experimental_apis = true
+
+deps += ['eal']
diff --git a/lib/librte_graph/rte_graph.h b/lib/librte_graph/rte_graph.h
new file mode 100644
index 000000000..4bcf0a6e5
--- /dev/null
+++ b/lib/librte_graph/rte_graph.h
@@ -0,0 +1,786 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef _RTE_GRAPH_H_
+#define _RTE_GRAPH_H_
+
+/**
+ * @file rte_graph.h
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Graph architecture abstracts the data processing functions as
+ * "node" and "link" them together to create a complex "graph" to enable
+ * reusable/modular data processing functions.
+ *
+ * This API enables graph framework operations such as create, lookup,
+ * dump and destroy on graph and node operations such as clone,
+ * edge update, and edge shrink, etc. The API also allows to create the stats
+ * cluster to monitor per graph and per node stats.
+ *
+ */
+
+#include <stdbool.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define RTE_GRAPH_NAMESIZE 64 /**< Max length of graph name. */
+#define RTE_NODE_NAMESIZE 64 /**< Max length of node name. */
+#define RTE_GRAPH_OFF_INVALID UINT32_MAX /**< Invalid graph offset. */
+#define RTE_NODE_ID_INVALID UINT32_MAX /**< Invalid node id. */
+#define RTE_EDGE_ID_INVALID UINT16_MAX /**< Invalid edge id. */
+#define RTE_GRAPH_ID_INVALID UINT16_MAX /**< Invalid graph id. */
+#define RTE_GRAPH_FENCE 0xdeadbeef12345678ULL /**< Graph fence data. */
+
+typedef uint32_t rte_graph_off_t; /**< Graph offset type. */
+typedef uint32_t rte_node_t; /**< Node id type. */
+typedef uint16_t rte_edge_t; /**< Edge id type. */
+typedef uint16_t rte_graph_t; /**< Graph id type. */
+
+/** Burst size in terms of log2 */
+#if RTE_GRAPH_BURST_SIZE == 1
+#define RTE_GRAPH_BURST_SIZE_LOG2 0 /**< Object burst size of 1. */
+#elif RTE_GRAPH_BURST_SIZE == 2
+#define RTE_GRAPH_BURST_SIZE_LOG2 1 /**< Object burst size of 2. */
+#elif RTE_GRAPH_BURST_SIZE == 4
+#define RTE_GRAPH_BURST_SIZE_LOG2 2 /**< Object burst size of 4. */
+#elif RTE_GRAPH_BURST_SIZE == 8
+#define RTE_GRAPH_BURST_SIZE_LOG2 3 /**< Object burst size of 8. */
+#elif RTE_GRAPH_BURST_SIZE == 16
+#define RTE_GRAPH_BURST_SIZE_LOG2 4 /**< Object burst size of 16. */
+#elif RTE_GRAPH_BURST_SIZE == 32
+#define RTE_GRAPH_BURST_SIZE_LOG2 5 /**< Object burst size of 32. */
+#elif RTE_GRAPH_BURST_SIZE == 64
+#define RTE_GRAPH_BURST_SIZE_LOG2 6 /**< Object burst size of 64. */
+#elif RTE_GRAPH_BURST_SIZE == 128
+#define RTE_GRAPH_BURST_SIZE_LOG2 7 /**< Object burst size of 128. */
+#elif RTE_GRAPH_BURST_SIZE == 256
+#define RTE_GRAPH_BURST_SIZE_LOG2 8 /**< Object burst size of 256. */
+#else
+#error "Unsupported burst size"
+#endif
+
+/* Forward declaration */
+struct rte_node; /**< Node data */
+struct rte_graph; /**< Graph data */
+struct rte_graph_cluster_stats; /**< Stats for Cluster of graphs */
+struct rte_graph_cluster_node_stats; /**< Node stats within cluster of graphs */
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node process function.
+ *
+ * The function invoked when the worker thread walks on nodes using
+ * rte_graph_walk().
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ * @param objs
+ * Pointer to an array of objects to be processed.
+ * @param nb_objs
+ * Number of objects in the array.
+ *
+ * @return
+ * Number of objects processed.
+ *
+ * @see rte_graph_walk()
+ *
+ */
+typedef uint16_t (*rte_node_process_t)(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node initialization function.
+ *
+ * The function invoked when the user creates the graph using rte_graph_create()
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ *
+ * @return
+ * - 0: Success.
+ * -<0: Failure.
+ *
+ * @see rte_graph_create()
+ */
+typedef int (*rte_node_init_t)(const struct rte_graph *graph,
+ struct rte_node *node);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node finalization function.
+ *
+ * The function invoked when the user destroys the graph using
+ * rte_graph_destroy().
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ *
+ * @see rte_graph_destroy()
+ */
+typedef void (*rte_node_fini_t)(const struct rte_graph *graph,
+ struct rte_node *node);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Graph cluster stats callback.
+ *
+ * @param is_first
+ * Flag to denote that stats are of the first node.
+ * @param is_last
+ * Flag to denote that stats are of the last node.
+ * @param cookie
+ * Cookie supplied during stats creation.
+ * @param stats
+ * Node cluster stats data.
+ *
+ * @return
+ * - 0: Success.
+ * -<0: Failure.
+ */
+typedef int (*rte_graph_cluster_stats_cb_t)(bool is_first, bool is_last,
+ void *cookie, const struct rte_graph_cluster_node_stats *stats);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Structure to hold configuration parameters for creating the graph.
+ *
+ * @see rte_graph_create()
+ */
+struct rte_graph_param {
+ int socket_id; /**< Socket id where memory is allocated. */
+ uint16_t nb_node_patterns; /**< Number of node patterns. */
+ const char **node_patterns;
+ /**< Array of node patterns based on shell pattern. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Structure to hold configuration parameters for graph cluster stats create.
+ *
+ * @see rte_graph_cluster_stats_create()
+ */
+struct rte_graph_cluster_stats_param {
+ int socket_id;
+ /**< Socket id where memory is allocated */
+ rte_graph_cluster_stats_cb_t fn;
+ /**< Stats print callback function. NULL value allowed, in that case,
+ * default print stat function used.
+ */
+ RTE_STD_C11
+ union {
+ void *cookie;
+ FILE *f; /**< File pointer to dump the stats when fn == NULL. */
+ };
+ uint16_t nb_graph_patterns; /**< Number of graph patterns. */
+ const char **graph_patterns;
+ /**< Array of graph patterns based on shell pattern. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node cluster stats data structure.
+ *
+ * @see struct rte_graph_cluster_stats_param::fn
+ */
+struct rte_graph_cluster_node_stats {
+ uint64_t ts; /**< Current timestamp. */
+ uint64_t calls; /**< Current number of calls made. */
+ uint64_t objs; /**< Current number of objs processed. */
+ uint64_t cycles; /**< Current number of cycles. */
+
+ uint64_t prev_ts; /**< Previous call timestamp. */
+ uint64_t prev_calls; /**< Previous number of calls. */
+ uint64_t prev_objs; /**< Previous number of processed objs. */
+ uint64_t prev_cycles; /**< Previous number of cycles. */
+
+ uint64_t realloc_count; /**< Realloc count. */
+
+ rte_node_t id; /**< Node identifier of stats. */
+ uint64_t hz; /**< Cycles per seconds. */
+ char name[RTE_NODE_NAMESIZE]; /**< Name of the node. */
+} __rte_cache_aligned;
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Structure defines the node registration parameters.
+ *
+ * @see __rte_node_register(), RTE_NODE_REGISTER()
+ */
+struct rte_node_register {
+ char name[RTE_NODE_NAMESIZE]; /**< Name of the node. */
+ uint64_t flags; /**< Node configuration flag. */
+#define RTE_NODE_SOURCE_F (1ULL << 0) /**< Node type is source. */
+ rte_node_process_t process; /**< Node process function. */
+ rte_node_init_t init; /**< Node init function. */
+ rte_node_fini_t fini; /**< Node fini function. */
+ rte_node_t id; /**< Node Identifier. */
+ rte_node_t parent_id; /**< Identifier of parent node. */
+ rte_edge_t nb_edges; /**< Number of edges from this node. */
+ const char *next_nodes[]; /**< Names of next nodes. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create Graph.
+ *
+ * Create memory reel, detect loops and find isolated nodes.
+ *
+ * @param name
+ * Unique name for this graph.
+ * @param prm
+ * Graph parameter, includes node names and count to be included
+ * in this graph.
+ *
+ * @return
+ * Unique graph id on success, RTE_GRAPH_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_create(const char *name, struct rte_graph_param *prm);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Destroy Graph.
+ *
+ * Free Graph memory reel.
+ *
+ * @param name
+ * Name of the graph to destroy.
+ *
+ * @return
+ * 0 on success, error otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_destroy(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get graph id from graph name.
+ *
+ * @param name
+ * Name of the graph to get id.
+ *
+ * @return
+ * Graph id on success, RTE_GRAPH_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_from_name(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get graph name from graph id.
+ *
+ * @param id
+ * id of the graph to get name.
+ *
+ * @return
+ * Graph name on success, NULL otherwise.
+ */
+__rte_experimental
+char *rte_graph_id_to_name(rte_graph_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Export the graph as graph viz dot file
+ *
+ * @param name
+ * Name of the graph to export.
+ * @param f
+ * File pointer to export the graph.
+ *
+ * @return
+ * 0 on success, error otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_export(const char *name, FILE *f);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get graph object from its name.
+ *
+ * Typical usage of this API to get graph objects in the worker thread and
+ * followed calling rte_graph_walk() in a loop.
+ *
+ * @param name
+ * Name of the graph.
+ *
+ * @return
+ * Graph pointer on success, NULL otherwise.
+ *
+ * @see rte_graph_walk()
+ */
+__rte_experimental
+struct rte_graph *rte_graph_lookup(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get maximum number of graph available.
+ *
+ * @return
+ * Maximum graph count on success, RTE_GRAPH_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_max_count(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump the graph information to file.
+ *
+ * @param f
+ * File pointer to dump graph info.
+ * @param id
+ * Graph id to get graph info.
+ */
+__rte_experimental
+void rte_graph_dump(FILE *f, rte_graph_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump all graphs information to file
+ *
+ * @param f
+ * File pointer to dump graph info.
+ */
+__rte_experimental
+void rte_graph_list_dump(FILE *f);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump graph information along with node info to file
+ *
+ * @param f
+ * File pointer to dump graph info.
+ * @param graph
+ * Graph pointer to get graph info.
+ * @param all
+ * true to dump nodes in the graph.
+ */
+__rte_experimental
+void rte_graph_obj_dump(FILE *f, struct rte_graph *graph, bool all);
+
+/** Macro to browse rte_node object after the graph creation */
+#define rte_graph_foreach_node(count, off, graph, node) \
+ for (count = 0, off = graph->nodes_start, \
+ node = RTE_PTR_ADD(graph, off); \
+ count < graph->nb_nodes; \
+ off = node->next, node = RTE_PTR_ADD(graph, off), count++)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get node object with in graph from id.
+ *
+ * @param graph_id
+ * Graph id to get node pointer from.
+ * @param node_id
+ * Node id to get node pointer.
+ *
+ * @return
+ * Node pointer on success, NULL otherwise.
+ */
+__rte_experimental
+struct rte_node *rte_graph_node_get(rte_graph_t graph_id, uint32_t node_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get node pointer with in graph from name.
+ *
+ * @param graph
+ * Graph name to get node pointer from.
+ * @param name
+ * Node name to get the node pointer.
+ *
+ * @return
+ * Node pointer on success, NULL otherwise.
+ */
+__rte_experimental
+struct rte_node *rte_graph_node_get_by_name(const char *graph,
+ const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create graph stats cluster to aggregate runtime node stats.
+ *
+ * @param prm
+ * Parameters including file pointer to dump stats,
+ * Graph pattern to create cluster and callback function.
+ *
+ * @return
+ * Valid pointer on success, NULL otherwise.
+ */
+__rte_experimental
+struct rte_graph_cluster_stats *rte_graph_cluster_stats_create(
+ const struct rte_graph_cluster_stats_param *prm);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Destroy cluster stats.
+ *
+ * @param stat
+ * Valid cluster pointer to destroy.
+ *
+ */
+__rte_experimental
+void rte_graph_cluster_stats_destroy(struct rte_graph_cluster_stats *stat);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get stats to application.
+ *
+ * @param[out] stat
+ * Cluster status.
+ * @param skip_cb
+ * true to skip callback function invocation.
+ */
+__rte_experimental
+void rte_graph_cluster_stats_get(struct rte_graph_cluster_stats *stat,
+ bool skip_cb);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Reset cluster stats to zero.
+ *
+ * @param stat
+ * Valid cluster stats pointer.
+ */
+__rte_experimental
+void rte_graph_cluster_stats_reset(struct rte_graph_cluster_stats *stat);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register new packet processing node. Nodes can be registered
+ * dynamically via this call or statically via the RTE_NODE_REGISTER
+ * macro.
+ *
+ * @param node
+ * Valid node pointer with name, process function and next_nodes.
+ *
+ * @return
+ * Valid node id on success, RTE_NODE_ID_INVALID otherwise.
+ *
+ * @see RTE_NODE_REGISTER()
+ */
+__rte_experimental
+rte_node_t __rte_node_register(const struct rte_node_register *node);
+
+/**
+ * Register a static node.
+ *
+ * The static node is registered through the constructor scheme, thereby, it can
+ * be used in a multi-process scenario.
+ *
+ * @param node
+ * Valid node pointer with name, process function, and next_nodes.
+ */
+#define RTE_NODE_REGISTER(node) \
+ RTE_INIT(rte_node_register_##node) \
+ { \
+ node.parent_id = RTE_NODE_ID_INVALID; \
+ node.id = __rte_node_register(&node); \
+ }
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Clone a node from static node(node created from RTE_NODE_REGISTER).
+ *
+ * @param id
+ * Static node id to clone from.
+ * @param name
+ * Name of the new node. The library prepends the parent node name to the
+ * user-specified the name. The final node name will be,
+ * "parent node name" + "-" + name.
+ *
+ * @return
+ * Valid node id on success, RTE_NODE_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_node_t rte_node_clone(rte_node_t id, const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get node id from node name.
+ *
+ * @param name
+ * Valid node name. In the case of the cloned node, the name will be
+ * "parent node name" + "-" + name.
+ *
+ * @return
+ * Valid node id on success, RTE_NODE_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_node_t rte_node_from_name(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get node name from node id.
+ *
+ * @param id
+ * Valid node id.
+ *
+ * @return
+ * Valid node name on success, NULL otherwise.
+ */
+__rte_experimental
+char *rte_node_id_to_name(rte_node_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get the number of edges for a node from node id.
+ *
+ * @param id
+ * Valid node id.
+ *
+ * @return
+ * Valid edge count on success, RTE_EDGE_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_edge_t rte_node_edge_count(rte_node_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Update the edges for a node from node id.
+ *
+ * @param id
+ * Valid node id.
+ * @param from
+ * Index to update the edges from. RTE_EDGE_ID_INVALID is valid,
+ * in that case, it will be added to the end of the list.
+ * @param next_nodes
+ * Name of the edges to update.
+ * @param nb_edges
+ * Number of edges to update.
+ *
+ * @return
+ * Valid edge count on success, 0 otherwise.
+ */
+__rte_experimental
+rte_edge_t rte_node_edge_update(rte_node_t id, rte_edge_t from,
+ const char **next_nodes, uint16_t nb_edges);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Shrink the edges to a given size.
+ *
+ * @param id
+ * Valid node id.
+ * @param size
+ * New size to shrink the edges.
+ *
+ * @return
+ * New size on success, RTE_EDGE_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_edge_t rte_node_edge_shrink(rte_node_t id, rte_edge_t size);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get the edge names from a given node.
+ *
+ * @param id
+ * Valid node id.
+ * @param[out] next_nodes
+ * Buffer to copy the edge names. The NULL value is allowed in that case,
+ * the function returns the size of the array that needs to be allocated.
+ *
+ * @return
+ * When next_nodes == NULL, it returns the size of the array else
+ * number of item copied.
+ */
+__rte_experimental
+rte_node_t rte_node_edge_get(rte_node_t id, char *next_nodes[]);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get maximum nodes created so far.
+ *
+ * @return
+ * Maximum nodes count on success, 0 otherwise.
+ */
+__rte_experimental
+rte_node_t rte_node_max_count(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump node info to file.
+ *
+ * @param f
+ * File pointer to dump the node info.
+ * @param id
+ * Node id to get the info.
+ */
+__rte_experimental
+void rte_node_dump(FILE *f, rte_node_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump all node info to file.
+ *
+ * @param f
+ * File pointer to dump the node info.
+ */
+__rte_experimental
+void rte_node_list_dump(FILE *f);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Test the validity of node id.
+ *
+ * @param id
+ * Node id to check.
+ *
+ * @return
+ * 1 if valid id, 0 otherwise.
+ */
+static __rte_always_inline int
+rte_node_is_invalid(rte_node_t id)
+{
+ return (id == RTE_NODE_ID_INVALID);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Test the validity of edge id.
+ *
+ * @param id
+ * Edge node id to check.
+ *
+ * @return
+ * 1 if valid id, 0 otherwise.
+ */
+static __rte_always_inline int
+rte_edge_is_invalid(rte_edge_t id)
+{
+ return (id == RTE_EDGE_ID_INVALID);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Test the validity of graph id.
+ *
+ * @param id
+ * Graph id to check.
+ *
+ * @return
+ * 1 if valid id, 0 otherwise.
+ */
+static __rte_always_inline int
+rte_graph_is_invalid(rte_graph_t id)
+{
+ return (id == RTE_GRAPH_ID_INVALID);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Test stats feature support.
+ *
+ * @return
+ * 1 if stats enabled, 0 otherwise.
+ */
+static __rte_always_inline int
+rte_graph_has_stats_feature(void)
+{
+#ifdef RTE_LIBRTE_GRAPH_STATS
+ return RTE_LIBRTE_GRAPH_STATS;
+#else
+ return 0;
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_GRAPH_H_ */
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
new file mode 100644
index 000000000..55ef35df5
--- /dev/null
+++ b/lib/librte_graph/rte_graph_version.map
@@ -0,0 +1,3 @@
+EXPERIMENTAL {
+
+};
diff --git a/lib/meson.build b/lib/meson.build
index 9c3cc55d5..c43d86bb9 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -30,7 +30,7 @@ libraries = [
# add pkt framework libs which use other libs from above
'port', 'table', 'pipeline',
# flow_classify lib depends on pkt framework table lib
- 'flow_classify', 'bpf', 'telemetry']
+ 'flow_classify', 'bpf', 'graph', 'telemetry']
if is_windows
libraries = ['kvargs','eal'] # only supported libraries for windows
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..b1195f09a 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -98,6 +98,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
_LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
_LDLIBS-$(CONFIG_RTE_LIBRTE_RCU) += -lrte_rcu
+_LDLIBS-$(CONFIG_RTE_LIBRTE_GRAPH) += -lrte_graph
ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_KNI) += -lrte_kni
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 02/26] graph: implement node registration
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 01/26] graph: define the public API for graph support jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 03/26] graph: implement node operations jerinj
` (24 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding rte_node_register() API implementation includes allocating
memory for node object, check for duplicate node name and
add the allocated node to STAILQ node_list for future use.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/Makefile | 1 +
lib/librte_graph/graph.c | 18 +++-
lib/librte_graph/graph_private.h | 74 ++++++++++++++++
lib/librte_graph/meson.build | 2 +-
lib/librte_graph/node.c | 115 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 4 +
6 files changed, 212 insertions(+), 2 deletions(-)
create mode 100644 lib/librte_graph/graph_private.h
create mode 100644 lib/librte_graph/node.c
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 26fe514f3..933d0ee49 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -14,6 +14,7 @@ LDLIBS += -lrte_eal
EXPORT_MAP := rte_graph_version.map
# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
# install header files
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index a55bf443a..a9c124896 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -2,4 +2,20 @@
* Copyright(C) 2020 Marvell International Ltd.
*/
-#include "rte_graph.h"
+#include <rte_spinlock.h>
+
+#include "graph_private.h"
+
+static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
+
+void
+graph_spinlock_lock(void)
+{
+ rte_spinlock_lock(&graph_lock);
+}
+
+void
+graph_spinlock_unlock(void)
+{
+ rte_spinlock_unlock(&graph_lock);
+}
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
new file mode 100644
index 000000000..ba56927fa
--- /dev/null
+++ b/lib/librte_graph/graph_private.h
@@ -0,0 +1,74 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef _RTE_GRAPH_PRIVATE_H_
+#define _RTE_GRAPH_PRIVATE_H_
+
+#include <inttypes.h>
+#include <sys/queue.h>
+
+#include <rte_common.h>
+#include <rte_eal.h>
+#include <rte_graph.h>
+
+/**
+ * @internal
+ *
+ * Structure that holds node internal data.
+ */
+struct node {
+ STAILQ_ENTRY(node) next; /**< Next node in the list. */
+ char name[RTE_NODE_NAMESIZE]; /**< Name of the node. */
+ uint64_t flags; /**< Node configuration flag. */
+ rte_node_process_t process; /**< Node process function. */
+ rte_node_init_t init; /**< Node init function. */
+ rte_node_fini_t fini; /**< Node fini function. */
+ rte_node_t id; /**< Allocated identifier for the node. */
+ rte_node_t parent_id; /**< Parent node identifier. */
+ rte_edge_t nb_edges; /**< Number of edges from this node. */
+ char next_nodes[][RTE_NODE_NAMESIZE]; /**< Names of next nodes. */
+};
+
+/* Node functions */
+STAILQ_HEAD(node_head, node);
+
+/**
+ * @internal
+ *
+ * Get the head of the node list.
+ *
+ * @return
+ * Pointer to the node head.
+ */
+struct node_head *node_list_head_get(void);
+
+/**
+ * @internal
+ *
+ * Get node pointer from node name.
+ *
+ * @param name
+ * Pointer to character string containing the node name.
+ *
+ * @return
+ * Pointer to the node.
+ */
+struct node *node_from_name(const char *name);
+
+/* Lock functions */
+/**
+ * @internal
+ *
+ * Take a lock on the graph internal spin lock.
+ */
+void graph_spinlock_lock(void);
+
+/**
+ * @internal
+ *
+ * Release a lock on the graph internal spin lock.
+ */
+void graph_spinlock_unlock(void);
+
+#endif /* _RTE_GRAPH_PRIVATE_H_ */
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index 455cf2ba5..5754ac23b 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,7 +4,7 @@
name = 'graph'
-sources = files('graph.c')
+sources = files('node.c', 'graph.c')
headers = files('rte_graph.h')
allow_experimental_apis = true
diff --git a/lib/librte_graph/node.c b/lib/librte_graph/node.c
new file mode 100644
index 000000000..7999ca6ed
--- /dev/null
+++ b/lib/librte_graph/node.c
@@ -0,0 +1,115 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_eal.h>
+#include <rte_errno.h>
+#include <rte_string_fns.h>
+
+#include "graph_private.h"
+
+static struct node_head node_list = STAILQ_HEAD_INITIALIZER(node_list);
+static rte_node_t node_id;
+
+#define NODE_ID_CHECK(id) ID_CHECK(id, node_id)
+
+/* Private functions */
+struct node_head *
+node_list_head_get(void)
+{
+ return &node_list;
+}
+
+struct node *
+node_from_name(const char *name)
+{
+ struct node *node;
+
+ STAILQ_FOREACH(node, &node_list, next)
+ if (strncmp(node->name, name, RTE_NODE_NAMESIZE) == 0)
+ return node;
+
+ return NULL;
+}
+
+static bool
+node_has_duplicate_entry(const char *name)
+{
+ struct node *node;
+
+ /* Is duplicate name registered */
+ STAILQ_FOREACH(node, &node_list, next) {
+ if (strncmp(node->name, name, RTE_NODE_NAMESIZE) == 0) {
+ rte_errno = EEXIST;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+/* Public functions */
+rte_node_t
+__rte_node_register(const struct rte_node_register *reg)
+{
+ struct node *node;
+ rte_edge_t i;
+ size_t sz;
+
+ graph_spinlock_lock();
+
+ /* Check sanity */
+ if (reg == NULL || reg->process == NULL) {
+ rte_errno = EINVAL;
+ goto fail;
+ }
+
+ /* Check for duplicate name */
+ if (node_has_duplicate_entry(reg->name))
+ goto fail;
+
+ sz = sizeof(struct node) + (reg->nb_edges * RTE_NODE_NAMESIZE);
+ node = calloc(1, sz);
+ if (node == NULL) {
+ rte_errno = ENOMEM;
+ goto fail;
+ }
+
+ /* Initialize the node */
+ if (rte_strscpy(node->name, reg->name, RTE_NODE_NAMESIZE) < 0) {
+ rte_errno = E2BIG;
+ goto free;
+ }
+ node->flags = reg->flags;
+ node->process = reg->process;
+ node->init = reg->init;
+ node->fini = reg->fini;
+ node->nb_edges = reg->nb_edges;
+ node->parent_id = reg->parent_id;
+ for (i = 0; i < reg->nb_edges; i++) {
+ if (rte_strscpy(node->next_nodes[i], reg->next_nodes[i],
+ RTE_NODE_NAMESIZE) < 0) {
+ rte_errno = E2BIG;
+ goto free;
+ }
+ }
+
+ node->id = node_id++;
+
+ /* Add the node at tail */
+ STAILQ_INSERT_TAIL(&node_list, node, next);
+ graph_spinlock_unlock();
+
+ return node->id;
+free:
+ free(node);
+fail:
+ graph_spinlock_unlock();
+ return RTE_NODE_ID_INVALID;
+}
+
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 55ef35df5..0884c09f1 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -1,3 +1,7 @@
EXPERIMENTAL {
+ global:
+ __rte_node_register;
+
+ local: *;
};
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 03/26] graph: implement node operations
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 01/26] graph: define the public API for graph support jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 02/26] graph: implement node registration jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 04/26] graph: implement node debug routines jerinj
` (23 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding node-specific API implementation like cloning node, updating
edges for the node, shrinking edges of a node, retrieving edges of a
node.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph_private.h | 10 +
lib/librte_graph/node.c | 269 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 10 +
3 files changed, 289 insertions(+)
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index ba56927fa..28405ddb7 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -12,6 +12,16 @@
#include <rte_eal.h>
#include <rte_graph.h>
+
+#define ID_CHECK(id, id_max) \
+ do { \
+ if ((id) >= (id_max)) { \
+ rte_errno = EINVAL; \
+ goto fail; \
+ } \
+ } while (0)
+
+
/**
* @internal
*
diff --git a/lib/librte_graph/node.c b/lib/librte_graph/node.c
index 7999ca6ed..8de857889 100644
--- a/lib/librte_graph/node.c
+++ b/lib/librte_graph/node.c
@@ -113,3 +113,272 @@ __rte_node_register(const struct rte_node_register *reg)
return RTE_NODE_ID_INVALID;
}
+static int
+clone_name(struct rte_node_register *reg, struct node *node, const char *name)
+{
+ ssize_t sz, rc;
+
+#define SZ RTE_NODE_NAMESIZE
+ rc = rte_strscpy(reg->name, node->name, SZ);
+ if (rc < 0)
+ goto fail;
+ sz = rc;
+ rc = rte_strscpy(reg->name + sz, "-", RTE_MAX((int16_t)(SZ - sz), 0));
+ if (rc < 0)
+ goto fail;
+ sz += rc;
+ sz = rte_strscpy(reg->name + sz, name, RTE_MAX((int16_t)(SZ - sz), 0));
+ if (sz < 0)
+ goto fail;
+
+ return 0;
+fail:
+ rte_errno = E2BIG;
+ return -rte_errno;
+}
+
+static rte_node_t
+node_clone(struct node *node, const char *name)
+{
+ rte_node_t rc = RTE_NODE_ID_INVALID;
+ struct rte_node_register *reg;
+ rte_edge_t i;
+
+ /* Don't allow to clone a node from a cloned node */
+ if (node->parent_id != RTE_NODE_ID_INVALID) {
+ rte_errno = EEXIST;
+ goto fail;
+ }
+
+ /* Check for duplicate name */
+ if (node_has_duplicate_entry(name))
+ goto fail;
+
+ reg = calloc(1, sizeof(*reg) + (sizeof(char *) * node->nb_edges));
+ if (reg == NULL) {
+ rte_errno = ENOMEM;
+ goto fail;
+ }
+
+ /* Clone the source node */
+ reg->flags = node->flags;
+ reg->process = node->process;
+ reg->init = node->init;
+ reg->fini = node->fini;
+ reg->nb_edges = node->nb_edges;
+ reg->parent_id = node->id;
+
+ for (i = 0; i < node->nb_edges; i++)
+ reg->next_nodes[i] = node->next_nodes[i];
+
+ /* Naming ceremony of the new node. name is node->name + "-" + name */
+ if (clone_name(reg, node, name))
+ goto free;
+
+ rc = __rte_node_register(reg);
+free:
+ free(reg);
+fail:
+ return rc;
+}
+
+rte_node_t
+rte_node_clone(rte_node_t id, const char *name)
+{
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ STAILQ_FOREACH(node, &node_list, next)
+ if (node->id == id)
+ return node_clone(node, name);
+
+fail:
+ return RTE_NODE_ID_INVALID;
+}
+
+rte_node_t
+rte_node_from_name(const char *name)
+{
+ struct node *node;
+
+ STAILQ_FOREACH(node, &node_list, next)
+ if (strncmp(node->name, name, RTE_NODE_NAMESIZE) == 0)
+ return node->id;
+
+ return RTE_NODE_ID_INVALID;
+}
+
+char *
+rte_node_id_to_name(rte_node_t id)
+{
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ STAILQ_FOREACH(node, &node_list, next)
+ if (node->id == id)
+ return node->name;
+
+fail:
+ return NULL;
+}
+
+rte_edge_t
+rte_node_edge_count(rte_node_t id)
+{
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ STAILQ_FOREACH(node, &node_list, next)
+ if (node->id == id)
+ return node->nb_edges;
+fail:
+ return RTE_EDGE_ID_INVALID;
+}
+
+static rte_edge_t
+edge_update(struct node *node, struct node *prev, rte_edge_t from,
+ const char **next_nodes, rte_edge_t nb_edges)
+{
+ rte_edge_t i, max_edges, count = 0;
+ struct node *new_node;
+ bool need_realloc;
+ size_t sz;
+
+ if (from == RTE_EDGE_ID_INVALID)
+ from = node->nb_edges;
+
+ /* Don't create hole in next_nodes[] list */
+ if (from > node->nb_edges) {
+ rte_errno = ENOMEM;
+ goto fail;
+ }
+
+ /* Remove me from list */
+ STAILQ_REMOVE(&node_list, node, node, next);
+
+ /* Allocate the storage space for new node if required */
+ max_edges = from + nb_edges;
+ need_realloc = max_edges > node->nb_edges;
+ if (need_realloc) {
+ sz = sizeof(struct node) + (max_edges * RTE_NODE_NAMESIZE);
+ new_node = realloc(node, sz);
+ if (new_node == NULL) {
+ rte_errno = ENOMEM;
+ goto restore;
+ } else {
+ node = new_node;
+ }
+ }
+
+ /* Update the new nodes name */
+ for (i = from; i < max_edges; i++, count++) {
+ if (rte_strscpy(node->next_nodes[i], next_nodes[count],
+ RTE_NODE_NAMESIZE) < 0) {
+ rte_errno = E2BIG;
+ goto restore;
+ }
+ }
+restore:
+ /* Update the linked list to point new node address in prev node */
+ if (prev)
+ STAILQ_INSERT_AFTER(&node_list, prev, node, next);
+ else
+ STAILQ_INSERT_HEAD(&node_list, node, next);
+
+ if (need_realloc)
+ node->nb_edges += count;
+
+fail:
+ return count;
+}
+
+rte_edge_t
+rte_node_edge_shrink(rte_node_t id, rte_edge_t size)
+{
+ rte_edge_t rc = RTE_EDGE_ID_INVALID;
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ graph_spinlock_lock();
+
+ STAILQ_FOREACH(node, &node_list, next) {
+ if (node->id == id) {
+ if (node->nb_edges < size) {
+ rte_errno = E2BIG;
+ goto fail;
+ }
+ node->nb_edges = size;
+ rc = size;
+ break;
+ }
+ }
+
+fail:
+ graph_spinlock_unlock();
+ return rc;
+}
+
+rte_edge_t
+rte_node_edge_update(rte_node_t id, rte_edge_t from, const char **next_nodes,
+ uint16_t nb_edges)
+{
+ rte_edge_t rc = RTE_EDGE_ID_INVALID;
+ struct node *n, *prev;
+
+ NODE_ID_CHECK(id);
+ graph_spinlock_lock();
+
+ prev = NULL;
+ STAILQ_FOREACH(n, &node_list, next) {
+ if (n->id == id) {
+ rc = edge_update(n, prev, from, next_nodes, nb_edges);
+ break;
+ }
+ prev = n;
+ }
+
+ graph_spinlock_unlock();
+fail:
+ return rc;
+}
+
+static rte_node_t
+node_copy_edges(struct node *node, char *next_nodes[])
+{
+ rte_edge_t i;
+
+ for (i = 0; i < node->nb_edges; i++)
+ next_nodes[i] = node->next_nodes[i];
+
+ return i;
+}
+
+rte_node_t
+rte_node_edge_get(rte_node_t id, char *next_nodes[])
+{
+ rte_node_t rc = RTE_NODE_ID_INVALID;
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ graph_spinlock_lock();
+
+ STAILQ_FOREACH(node, &node_list, next) {
+ if (node->id == id) {
+ if (next_nodes == NULL)
+ rc = sizeof(char *) * node->nb_edges;
+ else
+ rc = node_copy_edges(node, next_nodes);
+ break;
+ }
+ }
+
+ graph_spinlock_unlock();
+fail:
+ return rc;
+}
+
+rte_node_t
+rte_node_max_count(void)
+{
+ return node_id;
+}
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 0884c09f1..412386356 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -3,5 +3,15 @@ EXPERIMENTAL {
__rte_node_register;
+ rte_node_clone;
+ rte_node_edge_count;
+ rte_node_edge_get;
+ rte_node_edge_shrink;
+ rte_node_edge_update;
+ rte_node_from_name;
+ rte_node_id_to_name;
+ rte_node_list_dump;
+ rte_node_max_count;
+
local: *;
};
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 04/26] graph: implement node debug routines
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (2 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 03/26] graph: implement node operations jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 05/26] graph: implement internal graph operation helpers jerinj
` (22 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding node debug API implementation support to dump
single or all the node objects to the given file.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/Makefile | 1 +
lib/librte_graph/graph_debug.c | 25 ++++++++++++++++++++
lib/librte_graph/graph_private.h | 12 ++++++++++
lib/librte_graph/meson.build | 2 +-
lib/librte_graph/node.c | 32 ++++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 1 +
6 files changed, 72 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_graph/graph_debug.c
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 933d0ee49..2a6d86933 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -16,6 +16,7 @@ EXPORT_MAP := rte_graph_version.map
# all source are stored in SRCS-y
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
# install header files
SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
diff --git a/lib/librte_graph/graph_debug.c b/lib/librte_graph/graph_debug.c
new file mode 100644
index 000000000..75238e7ca
--- /dev/null
+++ b/lib/librte_graph/graph_debug.c
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_common.h>
+#include <rte_debug.h>
+
+#include "graph_private.h"
+
+void
+node_dump(FILE *f, struct node *n)
+{
+ rte_edge_t i;
+
+ fprintf(f, "node <%s>\n", n->name);
+ fprintf(f, " id=%" PRIu32 "\n", n->id);
+ fprintf(f, " flags=0x%" PRIx64 "\n", n->flags);
+ fprintf(f, " addr=%p\n", n);
+ fprintf(f, " process=%p\n", n->process);
+ fprintf(f, " nb_edges=%d\n", n->nb_edges);
+
+ for (i = 0; i < n->nb_edges; i++)
+ fprintf(f, " edge[%d] <%s>\n", i, n->next_nodes[i]);
+}
+
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index 28405ddb7..a8efee7c8 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -81,4 +81,16 @@ void graph_spinlock_lock(void);
*/
void graph_spinlock_unlock(void);
+/**
+ * @internal
+ *
+ * Dump internal node object data.
+ *
+ * @param f
+ * FILE pointer to dump the info.
+ * @param g
+ * Pointer to the internal node object.
+ */
+void node_dump(FILE *f, struct node *n);
+
#endif /* _RTE_GRAPH_PRIVATE_H_ */
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index 5754ac23b..01512182f 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,7 +4,7 @@
name = 'graph'
-sources = files('node.c', 'graph.c')
+sources = files('node.c', 'graph.c', 'graph_debug.c')
headers = files('rte_graph.h')
allow_experimental_apis = true
diff --git a/lib/librte_graph/node.c b/lib/librte_graph/node.c
index 8de857889..2f9c2ea4c 100644
--- a/lib/librte_graph/node.c
+++ b/lib/librte_graph/node.c
@@ -377,6 +377,38 @@ rte_node_edge_get(rte_node_t id, char *next_nodes[])
return rc;
}
+static void
+node_scan_dump(FILE *f, rte_node_t id, bool all)
+{
+ struct node *node;
+
+ RTE_ASSERT(f != NULL);
+ NODE_ID_CHECK(id);
+
+ STAILQ_FOREACH(node, &node_list, next) {
+ if (all == true) {
+ node_dump(f, node);
+ } else if (node->id == id) {
+ node_dump(f, node);
+ return;
+ }
+ }
+fail:
+ return;
+}
+
+void
+rte_node_dump(FILE *f, rte_node_t id)
+{
+ node_scan_dump(f, id, false);
+}
+
+void
+rte_node_list_dump(FILE *f)
+{
+ node_scan_dump(f, 0, true);
+}
+
rte_node_t
rte_node_max_count(void)
{
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 412386356..f2c2139c5 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -4,6 +4,7 @@ EXPERIMENTAL {
__rte_node_register;
rte_node_clone;
+ rte_node_dump;
rte_node_edge_count;
rte_node_edge_get;
rte_node_edge_shrink;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 05/26] graph: implement internal graph operation helpers
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (3 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 04/26] graph: implement node debug routines jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 06/26] graph: populate fastpath memory for graph reel jerinj
` (21 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding internal graph API helpers support to check whether a graph has
isolated nodes and any node have a loop to itself and BFS
algorithm implementation etc.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
---
lib/librte_graph/Makefile | 1 +
lib/librte_graph/graph.c | 2 +-
lib/librte_graph/graph_ops.c | 169 ++++++++++++++++++++++++++++++
lib/librte_graph/graph_private.h | 173 +++++++++++++++++++++++++++++++
lib/librte_graph/meson.build | 2 +-
5 files changed, 345 insertions(+), 2 deletions(-)
create mode 100644 lib/librte_graph/graph_ops.c
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 2a6d86933..39ecb2652 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -16,6 +16,7 @@ EXPORT_MAP := rte_graph_version.map
# all source are stored in SRCS-y
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
# install header files
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index a9c124896..4c3f2fe7b 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -7,7 +7,7 @@
#include "graph_private.h"
static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
-
+int rte_graph_logtype;
void
graph_spinlock_lock(void)
{
diff --git a/lib/librte_graph/graph_ops.c b/lib/librte_graph/graph_ops.c
new file mode 100644
index 000000000..335595311
--- /dev/null
+++ b/lib/librte_graph/graph_ops.c
@@ -0,0 +1,169 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <stdbool.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+
+#include "graph_private.h"
+
+/* Check whether a node has next_node to itself */
+static inline int
+node_has_loop_edge(struct node *node)
+{
+ rte_edge_t i;
+ char *name;
+ int rc = 0;
+
+ for (i = 0; i < node->nb_edges; i++) {
+ if (strncmp(node->name, node->next_nodes[i],
+ RTE_NODE_NAMESIZE) == 0) {
+ name = node->name;
+ rc = 1;
+ SET_ERR_JMP(EINVAL, fail, "Node %s has loop to self",
+ name);
+ }
+ }
+fail:
+ return rc;
+}
+
+int
+graph_node_has_loop_edge(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (node_has_loop_edge(graph_node->node))
+ return 1;
+
+ return 0;
+}
+
+rte_node_t
+graph_src_nodes_count(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ rte_node_t rc = 0;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (graph_node->node->flags & RTE_NODE_SOURCE_F)
+ rc++;
+
+ if (rc == 0)
+ SET_ERR_JMP(EINVAL, fail, "Graph needs at least a source node");
+fail:
+ return rc;
+}
+
+/* Check whether a node has next_node to a source node */
+int
+graph_node_has_edge_to_src_node(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ struct node *node;
+ rte_edge_t i;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ for (i = 0; i < graph_node->node->nb_edges; i++) {
+ node = graph_node->adjacency_list[i]->node;
+ if (node->flags & RTE_NODE_SOURCE_F)
+ SET_ERR_JMP(
+ EEXIST, fail,
+ "Node %s points to the source node %s",
+ graph_node->node->name, node->name);
+ }
+ }
+
+ return 0;
+fail:
+ return 1;
+}
+
+rte_node_t
+graph_nodes_count(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ rte_node_t count = 0;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ count++;
+
+ return count;
+}
+
+void
+graph_mark_nodes_as_not_visited(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ graph_node->visited = false;
+}
+
+int
+graph_bfs(struct graph *graph, struct graph_node *start)
+{
+ struct graph_node **queue, *v, *tmp;
+ uint16_t head = 0, tail = 0;
+ rte_edge_t i;
+ size_t sz;
+
+ sz = sizeof(struct graph_node *) * graph_nodes_count(graph);
+ queue = malloc(sz);
+ if (queue == NULL)
+ SET_ERR_JMP(ENOMEM, fail, "Failed to alloc BFS queue of %zu",
+ sz);
+
+ /* BFS algorithm */
+ queue[tail++] = start;
+ start->visited = true;
+ while (head != tail) {
+ v = queue[head++];
+ for (i = 0; i < v->node->nb_edges; i++) {
+ tmp = v->adjacency_list[i];
+ if (tmp->visited == false) {
+ queue[tail++] = tmp;
+ tmp->visited = true;
+ }
+ }
+ }
+
+ free(queue);
+ return 0;
+
+fail:
+ return -rte_errno;
+}
+
+/* Check whether a node has connected path or parent node */
+int
+graph_has_isolated_node(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ graph_mark_nodes_as_not_visited(graph);
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ if (graph_node->node->flags & RTE_NODE_SOURCE_F) {
+ if (graph_node->node->nb_edges == 0)
+ SET_ERR_JMP(EINVAL, fail,
+ "%s node needs minimum one edge",
+ graph_node->node->name);
+ if (graph_bfs(graph, graph_node))
+ goto fail;
+ }
+ }
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (graph_node->visited == false)
+ SET_ERR_JMP(EINVAL, fail, "Found isolated node %s",
+ graph_node->node->name);
+
+ return 0;
+fail:
+ return 1;
+}
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index a8efee7c8..7bf491d3d 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -12,6 +12,16 @@
#include <rte_eal.h>
#include <rte_graph.h>
+extern int rte_graph_logtype;
+
+#define GRAPH_LOG(level, ...) \
+ rte_log(RTE_LOG_##level, rte_graph_logtype, \
+ RTE_FMT("GRAPH: %s():%u " RTE_FMT_HEAD(__VA_ARGS__, ) "\n", \
+ __func__, __LINE__, RTE_FMT_TAIL(__VA_ARGS__, )))
+
+#define graph_err(...) GRAPH_LOG(ERR, __VA_ARGS__)
+#define graph_info(...) GRAPH_LOG(INFO, __VA_ARGS__)
+#define graph_dbg(...) GRAPH_LOG(DEBUG, __VA_ARGS__)
#define ID_CHECK(id, id_max) \
do { \
@@ -21,6 +31,12 @@
} \
} while (0)
+#define SET_ERR_JMP(err, where, fmt, ...) \
+ do { \
+ graph_err(fmt, ##__VA_ARGS__); \
+ rte_errno = err; \
+ goto where; \
+ } while (0)
/**
* @internal
@@ -40,6 +56,52 @@ struct node {
char next_nodes[][RTE_NODE_NAMESIZE]; /**< Names of next nodes. */
};
+/**
+ * @internal
+ *
+ * Structure that holds the graph node data.
+ */
+struct graph_node {
+ STAILQ_ENTRY(graph_node) next; /**< Next graph node in the list. */
+ struct node *node; /**< Pointer to internal node. */
+ bool visited; /**< Flag used in BFS to mark node visited. */
+ struct graph_node *adjacency_list[]; /**< Adjacency list of the node. */
+};
+
+/**
+ * @internal
+ *
+ * Structure that holds graph internal data.
+ */
+struct graph {
+ STAILQ_ENTRY(graph) next;
+ /**< List of graphs. */
+ char name[RTE_GRAPH_NAMESIZE];
+ /**< Name of the graph. */
+ const struct rte_memzone *mz;
+ /**< Memzone to store graph data. */
+ rte_graph_off_t nodes_start;
+ /**< Node memory start offset in graph reel. */
+ rte_node_t src_node_count;
+ /**< Number of source nodes in a graph. */
+ struct rte_graph *graph;
+ /**< Pointer to graph data. */
+ rte_node_t node_count;
+ /**< Total number of nodes. */
+ uint32_t cir_start;
+ /**< Circular buffer start offset in graph reel. */
+ uint32_t cir_mask;
+ /**< Circular buffer mask for wrap around. */
+ rte_graph_t id;
+ /**< Graph identifier. */
+ size_t mem_sz;
+ /**< Memory size of the graph. */
+ int socket;
+ /**< Socket identifier where memory is allocated. */
+ STAILQ_HEAD(gnode_list, graph_node) node_list;
+ /**< Nodes in a graph. */
+};
+
/* Node functions */
STAILQ_HEAD(node_head, node);
@@ -66,6 +128,19 @@ struct node_head *node_list_head_get(void);
*/
struct node *node_from_name(const char *name);
+/* Graph list functions */
+STAILQ_HEAD(graph_head, graph);
+
+/**
+ * @internal
+ *
+ * Get the head of the graph list.
+ *
+ * @return
+ * Pointer to the graph head.
+ */
+struct graph_head *graph_list_head_get(void);
+
/* Lock functions */
/**
* @internal
@@ -81,6 +156,104 @@ void graph_spinlock_lock(void);
*/
void graph_spinlock_unlock(void);
+/* Graph operations */
+/**
+ * @internal
+ *
+ * Run a BFS(Breadth First Search) on the graph marking all
+ * the graph nodes as visited.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ * @param start
+ * Pointer to the starting graph node.
+ *
+ * @return
+ * - 0: Success.
+ * - -ENOMEM: Not enough memory for BFS.
+ */
+int graph_bfs(struct graph *graph, struct graph_node *start);
+
+/**
+ * @internal
+ *
+ * Check if there is an isolated node in the given graph.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: No isolated node found.
+ * - 1: Isolated node found.
+ */
+int graph_has_isolated_node(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Check whether a node in the graph has next_node to a source node.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: Node has an edge to source node.
+ * - 1: Node doesn't have an edge to source node.
+ */
+int graph_node_has_edge_to_src_node(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Checks whether node in the graph has a edge to itself i.e. forms a
+ * loop.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: Node has an edge to itself.
+ * - 1: Node doesn't have an edge to itself.
+ */
+int graph_node_has_loop_edge(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Get the count of source nodes in the graph.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * Number of source nodes.
+ */
+rte_node_t graph_src_nodes_count(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Get the count of total number of nodes in the graph.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * Number of nodes.
+ */
+rte_node_t graph_nodes_count(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Clear the visited flag of all the nodes in the graph.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ */
+void graph_mark_nodes_as_not_visited(struct graph *graph);
+
+
/**
* @internal
*
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index 01512182f..16e0625c1 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,7 +4,7 @@
name = 'graph'
-sources = files('node.c', 'graph.c', 'graph_debug.c')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c')
headers = files('rte_graph.h')
allow_experimental_apis = true
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 06/26] graph: populate fastpath memory for graph reel
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (4 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 05/26] graph: implement internal graph operation helpers jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 07/26] graph: implement create and destroy APIs jerinj
` (20 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding support to create and populate the memory for graph reel.
This includes reserving the memory in the memzone, populating the nodes,
Allocating memory for node-specific streams to hold objects.
Once it is populated the reel memory contains the following sections.
+---------------------+
| Graph Header |
+---------------------+
| Fence |
+---------------------+
| Circular buffer |
+---------------------+
| Fence |
+---------------------+
| Node Object 0 |
+------------------- -+
| Node Object 1 |
+------------------- -+
| Node Object 2 |
+------------------- -+
| Node Object n |
+------------------- -+
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/Makefile | 2 +
lib/librte_graph/graph.c | 16 ++
lib/librte_graph/graph_populate.c | 234 +++++++++++++++++++++++++
lib/librte_graph/graph_private.h | 64 +++++++
lib/librte_graph/meson.build | 4 +-
lib/librte_graph/node.c | 5 +
lib/librte_graph/rte_graph_version.map | 1 +
lib/librte_graph/rte_graph_worker.h | 107 +++++++++++
8 files changed, 431 insertions(+), 2 deletions(-)
create mode 100644 lib/librte_graph/graph_populate.c
create mode 100644 lib/librte_graph/rte_graph_worker.h
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 39ecb2652..7bfd7d51f 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -18,8 +18,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_populate.c
# install header files
SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
+SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph_worker.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index 4c3f2fe7b..e1930b7d2 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -2,6 +2,7 @@
* Copyright(C) 2020 Marvell International Ltd.
*/
+#include <rte_malloc.h>
#include <rte_spinlock.h>
#include "graph_private.h"
@@ -19,3 +20,18 @@ graph_spinlock_unlock(void)
{
rte_spinlock_unlock(&graph_lock);
}
+
+void __rte_noinline
+__rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
+{
+ uint16_t size = node->size;
+
+ RTE_VERIFY(size != UINT16_MAX);
+ /* Allocate double amount of size to avoid immediate realloc */
+ size = RTE_MIN(UINT16_MAX, RTE_MAX(RTE_GRAPH_BURST_SIZE, size * 2));
+ node->objs = rte_realloc_socket(node->objs, size * sizeof(void *),
+ RTE_CACHE_LINE_SIZE, graph->socket);
+ RTE_VERIFY(node->objs);
+ node->size = size;
+ node->realloc_count++;
+}
diff --git a/lib/librte_graph/graph_populate.c b/lib/librte_graph/graph_populate.c
new file mode 100644
index 000000000..093512efa
--- /dev/null
+++ b/lib/librte_graph/graph_populate.c
@@ -0,0 +1,234 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <fnmatch.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+
+#include "graph_private.h"
+
+static size_t
+graph_fp_mem_calc_size(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ rte_node_t val;
+ size_t sz;
+
+ /* Graph header */
+ sz = sizeof(struct rte_graph);
+ /* Source nodes list */
+ sz += sizeof(rte_graph_off_t) * graph->src_node_count;
+ /* Circular buffer for pending streams of size number of nodes */
+ val = rte_align32pow2(graph->node_count * sizeof(rte_graph_off_t));
+ sz = RTE_ALIGN(sz, val);
+ graph->cir_start = sz;
+ graph->cir_mask = rte_align32pow2(graph->node_count) - 1;
+ sz += val;
+ /* Fence */
+ sz += sizeof(RTE_GRAPH_FENCE);
+ sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
+ graph->nodes_start = sz;
+ /* For 0..N node objects with fence */
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
+ sz += sizeof(struct rte_node);
+ /* Pointer to next nodes(edges) */
+ sz += sizeof(struct rte_node *) * graph_node->node->nb_edges;
+ }
+
+ graph->mem_sz = sz;
+ return sz;
+}
+
+static void
+graph_header_popluate(struct graph *_graph)
+{
+ struct rte_graph *graph = _graph->graph;
+
+ graph->tail = 0;
+ graph->head = (int32_t)-_graph->src_node_count;
+ graph->cir_mask = _graph->cir_mask;
+ graph->nb_nodes = _graph->node_count;
+ graph->cir_start = RTE_PTR_ADD(graph, _graph->cir_start);
+ graph->nodes_start = _graph->nodes_start;
+ graph->socket = _graph->socket;
+ graph->id = _graph->id;
+ memcpy(graph->name, _graph->name, RTE_GRAPH_NAMESIZE);
+ graph->fence = RTE_GRAPH_FENCE;
+}
+
+static void
+graph_nodes_populate(struct graph *_graph)
+{
+ rte_graph_off_t off = _graph->nodes_start;
+ struct rte_graph *graph = _graph->graph;
+ struct graph_node *graph_node;
+ rte_edge_t count, nb_edges;
+ const char *parent;
+ rte_node_t pid;
+
+ STAILQ_FOREACH(graph_node, &_graph->node_list, next) {
+ struct rte_node *node = RTE_PTR_ADD(graph, off);
+ memset(node, 0, sizeof(*node));
+ node->fence = RTE_GRAPH_FENCE;
+ node->off = off;
+ node->process = graph_node->node->process;
+ memcpy(node->name, graph_node->node->name, RTE_GRAPH_NAMESIZE);
+ pid = graph_node->node->parent_id;
+ if (pid != RTE_NODE_ID_INVALID) { /* Cloned node */
+ parent = rte_node_id_to_name(pid);
+ memcpy(node->parent, parent, RTE_GRAPH_NAMESIZE);
+ }
+ node->id = graph_node->node->id;
+ node->parent_id = pid;
+ nb_edges = graph_node->node->nb_edges;
+ node->nb_edges = nb_edges;
+ off += sizeof(struct rte_node);
+ /* Copy the name in first pass to replace with rte_node* later*/
+ for (count = 0; count < nb_edges; count++)
+ node->nodes[count] = (struct rte_node *)&graph_node
+ ->adjacency_list[count]
+ ->node->name[0];
+
+ off += sizeof(struct rte_node *) * nb_edges;
+ off = RTE_ALIGN(off, RTE_CACHE_LINE_SIZE);
+ node->next = off;
+ __rte_node_stream_alloc(graph, node);
+ }
+}
+
+struct rte_node *
+graph_node_id_to_ptr(const struct rte_graph *graph, rte_node_t id)
+{
+ rte_node_t count;
+ rte_graph_off_t off;
+ struct rte_node *node;
+
+ rte_graph_foreach_node(count, off, graph, node)
+ if (unlikely(node->id == id))
+ return node;
+
+ return NULL;
+}
+
+struct rte_node *
+graph_node_name_to_ptr(const struct rte_graph *graph, const char *name)
+{
+ rte_node_t count;
+ rte_graph_off_t off;
+ struct rte_node *node;
+
+ rte_graph_foreach_node(count, off, graph, node)
+ if (strncmp(name, node->name, RTE_NODE_NAMESIZE) == 0)
+ return node;
+
+ return NULL;
+}
+
+static int
+graph_node_nexts_populate(struct graph *_graph)
+{
+ rte_node_t count, val;
+ rte_graph_off_t off;
+ struct rte_node *node;
+ const struct rte_graph *graph = _graph->graph;
+ const char *name;
+
+ rte_graph_foreach_node(count, off, graph, node) {
+ for (val = 0; val < node->nb_edges; val++) {
+ name = (const char *)node->nodes[val];
+ node->nodes[val] = graph_node_name_to_ptr(graph, name);
+ if (node->nodes[val] == NULL)
+ SET_ERR_JMP(EINVAL, fail, "%s not found", name);
+ }
+ }
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static int
+graph_src_nodes_populate(struct graph *_graph)
+{
+ struct rte_graph *graph = _graph->graph;
+ struct graph_node *graph_node;
+ struct rte_node *node;
+ int32_t head = -1;
+ const char *name;
+
+ STAILQ_FOREACH(graph_node, &_graph->node_list, next) {
+ if (graph_node->node->flags & RTE_NODE_SOURCE_F) {
+ name = graph_node->node->name;
+ node = graph_node_name_to_ptr(graph, name);
+ if (node == NULL)
+ SET_ERR_JMP(EINVAL, fail, "%s not found", name);
+
+ __rte_node_stream_alloc(graph, node);
+ graph->cir_start[head--] = node->off;
+ }
+ }
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static int
+graph_fp_mem_populate(struct graph *graph)
+{
+ int rc;
+
+ graph_header_popluate(graph);
+ graph_nodes_populate(graph);
+ rc = graph_node_nexts_populate(graph);
+ rc |= graph_src_nodes_populate(graph);
+
+ return rc;
+}
+
+int
+graph_fp_mem_create(struct graph *graph)
+{
+ const struct rte_memzone *mz;
+ size_t sz;
+
+ sz = graph_fp_mem_calc_size(graph);
+ mz = rte_memzone_reserve(graph->name, sz, graph->socket, 0);
+ if (mz == NULL)
+ SET_ERR_JMP(ENOMEM, fail, "Memzone %s reserve failed",
+ graph->name);
+
+ graph->graph = mz->addr;
+ graph->mz = mz;
+
+ return graph_fp_mem_populate(graph);
+fail:
+ return -rte_errno;
+}
+
+static void
+graph_nodes_mem_destroy(struct rte_graph *graph)
+{
+ rte_node_t count;
+ rte_graph_off_t off;
+ struct rte_node *node;
+
+ if (graph == NULL)
+ return;
+
+ rte_graph_foreach_node(count, off, graph, node)
+ rte_free(node->objs);
+}
+
+int
+graph_fp_mem_destroy(struct graph *graph)
+{
+ graph_nodes_mem_destroy(graph->graph);
+ return rte_memzone_free(graph->mz);
+}
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index 7bf491d3d..051fad53a 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -11,6 +11,7 @@
#include <rte_common.h>
#include <rte_eal.h>
#include <rte_graph.h>
+#include <rte_graph_worker.h>
extern int rte_graph_logtype;
@@ -253,6 +254,69 @@ rte_node_t graph_nodes_count(struct graph *graph);
*/
void graph_mark_nodes_as_not_visited(struct graph *graph);
+/* Fast path graph memory populate unctions */
+
+/**
+ * @internal
+ *
+ * Create fast-path memory for the graph and nodes.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: Success.
+ * - -ENOMEM: Not enough for graph and nodes.
+ * - -EINVAL: Graph nodes not found.
+ */
+int graph_fp_mem_create(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Free fast-path memory used by graph and nodes.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: Success.
+ * - <0: Graph memzone related error.
+ */
+int graph_fp_mem_destroy(struct graph *graph);
+
+/* Lookup functions */
+/**
+ * @internal
+ *
+ * Get graph node object from node id.
+ *
+ * @param graph
+ * Pointer to rte_graph object.
+ * @param id
+ * Node Identifier.
+ *
+ * @return
+ * Pointer to rte_node if identifier is valid else NULL.
+ */
+struct rte_node *graph_node_id_to_ptr(const struct rte_graph *graph,
+ rte_node_t id);
+
+/**
+ * @internal
+ *
+ * Get graph node object from node name.
+ *
+ * @param graph
+ * Pointer to rte_graph object.
+ * @param node_name
+ * Pointer to character string holding the node name.
+ *
+ * @return
+ * Pointer to rte_node if identifier is valid else NULL.
+ */
+struct rte_node *graph_node_name_to_ptr(const struct rte_graph *graph,
+ const char *node_name);
/**
* @internal
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index 16e0625c1..fb203a5e2 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,8 +4,8 @@
name = 'graph'
-sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c')
-headers = files('rte_graph.h')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_populate.c')
+headers = files('rte_graph.h', 'rte_graph_worker.h')
allow_experimental_apis = true
deps += ['eal']
diff --git a/lib/librte_graph/node.c b/lib/librte_graph/node.c
index 2f9c2ea4c..639269870 100644
--- a/lib/librte_graph/node.c
+++ b/lib/librte_graph/node.c
@@ -61,6 +61,11 @@ __rte_node_register(const struct rte_node_register *reg)
rte_edge_t i;
size_t sz;
+ /* Limit Node specific metadata to one cacheline on 64B CL machine */
+ RTE_BUILD_BUG_ON((offsetof(struct rte_node, nodes) -
+ offsetof(struct rte_node, ctx)) !=
+ RTE_CACHE_LINE_MIN_SIZE);
+
graph_spinlock_lock();
/* Check sanity */
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index f2c2139c5..a9fe1b610 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -2,6 +2,7 @@ EXPERIMENTAL {
global:
__rte_node_register;
+ __rte_node_stream_alloc;
rte_node_clone;
rte_node_dump;
diff --git a/lib/librte_graph/rte_graph_worker.h b/lib/librte_graph/rte_graph_worker.h
new file mode 100644
index 000000000..a7c780d4d
--- /dev/null
+++ b/lib/librte_graph/rte_graph_worker.h
@@ -0,0 +1,107 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef _RTE_GRAPH_WORKER_H_
+#define _RTE_GRAPH_WORKER_H_
+
+/**
+ * @file rte_graph_worker.h
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * This API allows a worker thread to walk over a graph and nodes to create,
+ * process, enqueue and move streams of objects to the next nodes.
+ */
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_graph.h>
+#include <rte_prefetch.h>
+#include <rte_memcpy.h>
+#include <rte_memory.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * @internal
+ *
+ * Data structure to hold graph data.
+ */
+struct rte_graph {
+ uint32_t tail; /**< Tail of circular buffer. */
+ uint32_t head; /**< Head of circular buffer. */
+ uint32_t cir_mask; /**< Circular buffer wrap around mask. */
+ rte_node_t nb_nodes; /**< Number of nodes in the graph. */
+ rte_graph_off_t *cir_start; /**< Pointer to circular buffer. */
+ rte_graph_off_t nodes_start; /**< Offset at which node memory starts. */
+ rte_graph_t id; /**< Graph identifier. */
+ int socket; /**< Socket ID where memory is allocated. */
+ char name[RTE_GRAPH_NAMESIZE]; /**< Name of the graph. */
+ uint64_t fence; /**< Fence. */
+} __rte_cache_aligned;
+
+/**
+ * @internal
+ *
+ * Data structure to hold node data.
+ */
+struct rte_node {
+ /* Slow path area */
+ uint64_t fence; /**< Fence. */
+ rte_graph_off_t next; /**< Index to next node. */
+ rte_node_t id; /**< Node identifier. */
+ rte_node_t parent_id; /**< Parent Node identifier. */
+ rte_edge_t nb_edges; /**< Number of edges from this node. */
+ uint32_t realloc_count; /**< Number of times realloced. */
+
+ char parent[RTE_NODE_NAMESIZE]; /**< Parent node name. */
+ char name[RTE_NODE_NAMESIZE]; /**< Name of the node. */
+
+ /* Fast path area */
+#define RTE_NODE_CTX_SZ 16
+ uint8_t ctx[RTE_NODE_CTX_SZ] __rte_cache_aligned; /**< Node Context. */
+ uint16_t size; /**< Total number of objects available. */
+ uint16_t idx; /**< Number of objects used. */
+ rte_graph_off_t off; /**< Offset of node in the graph reel. */
+ uint64_t total_cycles; /**< Cycles spent in this node. */
+ uint64_t total_calls; /**< Calls done to this node. */
+ uint64_t total_objs; /**< Objects processed by this node. */
+ RTE_STD_C11
+ union {
+ void **objs; /**< Array of object pointers. */
+ uint64_t objs_u64;
+ };
+ RTE_STD_C11
+ union {
+ rte_node_process_t process; /**< Process function. */
+ uint64_t process_u64;
+ };
+ struct rte_node *nodes[] __rte_cache_min_aligned; /**< Next nodes. */
+} __rte_cache_aligned;
+
+/**
+ * @internal
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Allocate a stream of objects.
+ *
+ * If stream already exists then re-allocate it to a larger size.
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ */
+__rte_experimental
+void __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_GRAPH_WORKER_H_ */
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 07/26] graph: implement create and destroy APIs
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (5 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 06/26] graph: populate fastpath memory for graph reel jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 08/26] graph: implement graph operation APIs jerinj
` (19 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding graph specific API implementations like graph create
and graph destroy. This detect loops in the graph,
check for isolated nodes and operation to verify the validity of
graph.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph.c | 321 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 2 +
2 files changed, 323 insertions(+)
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index e1930b7d2..d060ffe70 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -2,13 +2,34 @@
* Copyright(C) 2020 Marvell International Ltd.
*/
+#include <fnmatch.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_errno.h>
+#include <rte_graph.h>
#include <rte_malloc.h>
+#include <rte_memzone.h>
#include <rte_spinlock.h>
+#include <rte_string_fns.h>
#include "graph_private.h"
+static struct graph_head graph_list = STAILQ_HEAD_INITIALIZER(graph_list);
static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
+static rte_graph_t graph_id;
int rte_graph_logtype;
+
+#define GRAPH_ID_CHECK(id) ID_CHECK(id, graph_id)
+
+/* Private functions */
+struct graph_head *
+graph_list_head_get(void)
+{
+ return &graph_list;
+}
+
void
graph_spinlock_lock(void)
{
@@ -21,6 +42,306 @@ graph_spinlock_unlock(void)
rte_spinlock_unlock(&graph_lock);
}
+static int
+graph_node_add(struct graph *graph, struct node *node)
+{
+ struct graph_node *graph_node;
+ size_t sz;
+
+ /* Skip the duplicate nodes */
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (strncmp(node->name, graph_node->node->name,
+ RTE_NODE_NAMESIZE) == 0)
+ return 0;
+
+ /* Allocate new graph node object */
+ sz = sizeof(*graph_node) + node->nb_edges * sizeof(struct node *);
+ graph_node = calloc(1, sz);
+
+ if (graph_node == NULL)
+ SET_ERR_JMP(ENOMEM, free, "Failed to calloc %s object",
+ node->name);
+
+ /* Initialize the graph node */
+ graph_node->node = node;
+
+ /* Add to graph node list */
+ STAILQ_INSERT_TAIL(&graph->node_list, graph_node, next);
+ return 0;
+
+free:
+ free(graph_node);
+ return -rte_errno;
+}
+
+static struct graph_node *
+node_to_graph_node(struct graph *graph, struct node *node)
+{
+ struct graph_node *graph_node;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (graph_node->node == node)
+ return graph_node;
+
+ SET_ERR_JMP(ENODEV, fail, "Found isolated node %s", node->name);
+fail:
+ return NULL;
+}
+
+static int
+graph_node_edges_add(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ struct node *adjacency;
+ const char *next;
+ rte_edge_t i;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ for (i = 0; i < graph_node->node->nb_edges; i++) {
+ next = graph_node->node->next_nodes[i];
+ adjacency = node_from_name(next);
+ if (adjacency == NULL)
+ SET_ERR_JMP(EINVAL, fail,
+ "Node %s not registered", next);
+ if (graph_node_add(graph, adjacency))
+ goto fail;
+ }
+ }
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static int
+graph_adjacency_list_update(struct graph *graph)
+{
+ struct graph_node *graph_node, *tmp;
+ struct node *adjacency;
+ const char *next;
+ rte_edge_t i;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ for (i = 0; i < graph_node->node->nb_edges; i++) {
+ next = graph_node->node->next_nodes[i];
+ adjacency = node_from_name(next);
+ if (adjacency == NULL)
+ SET_ERR_JMP(EINVAL, fail,
+ "Node %s not registered", next);
+ tmp = node_to_graph_node(graph, adjacency);
+ if (tmp == NULL)
+ goto fail;
+ graph_node->adjacency_list[i] = tmp;
+ }
+ }
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static int
+expand_pattern_to_node(struct graph *graph, const char *pattern)
+{
+ struct node_head *node_head = node_list_head_get();
+ bool found = false;
+ struct node *node;
+
+ /* Check for pattern match */
+ STAILQ_FOREACH(node, node_head, next) {
+ if (fnmatch(pattern, node->name, 0) == 0) {
+ if (graph_node_add(graph, node))
+ goto fail;
+ found = true;
+ }
+ }
+ if (found == false)
+ SET_ERR_JMP(EFAULT, fail, "Pattern %s node not found", pattern);
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static void
+graph_cleanup(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ while (!STAILQ_EMPTY(&graph->node_list)) {
+ graph_node = STAILQ_FIRST(&graph->node_list);
+ STAILQ_REMOVE_HEAD(&graph->node_list, next);
+ free(graph_node);
+ }
+}
+
+static int
+graph_node_init(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ const char *name;
+ int rc;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ if (graph_node->node->init) {
+ name = graph_node->node->name;
+ rc = graph_node->node->init(
+ graph->graph,
+ graph_node_name_to_ptr(graph->graph, name));
+ if (rc)
+ SET_ERR_JMP(rc, err, "Node %s init() failed",
+ name);
+ }
+ }
+
+ return 0;
+err:
+ return -rte_errno;
+}
+
+static void
+graph_node_fini(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (graph_node->node->fini)
+ graph_node->node->fini(
+ graph->graph,
+ graph_node_name_to_ptr(graph->graph,
+ graph_node->node->name));
+}
+
+rte_graph_t
+rte_graph_create(const char *name, struct rte_graph_param *prm)
+{
+ struct graph *graph;
+ const char *pattern;
+ uint16_t i;
+
+ graph_spinlock_lock();
+
+ /* Check arguments sanity */
+ if (prm == NULL)
+ SET_ERR_JMP(EINVAL, fail, "Param should not be NULL");
+
+ if (name == NULL)
+ SET_ERR_JMP(EINVAL, fail, "Graph name should not be NULL");
+
+ /* Check for existence of duplicate graph */
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (strncmp(name, graph->name, RTE_GRAPH_NAMESIZE) == 0)
+ SET_ERR_JMP(EEXIST, fail, "Found duplicate graph %s",
+ name);
+
+ /* Create graph object */
+ graph = calloc(1, sizeof(*graph));
+ if (graph == NULL)
+ SET_ERR_JMP(ENOMEM, fail, "Failed to calloc graph object");
+
+ /* Initialize the graph object */
+ STAILQ_INIT(&graph->node_list);
+ if (rte_strscpy(graph->name, name, RTE_GRAPH_NAMESIZE) < 0)
+ SET_ERR_JMP(E2BIG, free, "Too big name=%s", name);
+
+ /* Expand node pattern and add the nodes to the graph */
+ for (i = 0; i < prm->nb_node_patterns; i++) {
+ pattern = prm->node_patterns[i];
+ if (expand_pattern_to_node(graph, pattern))
+ goto graph_cleanup;
+ }
+
+ /* Go over all the nodes edges and add them to the graph */
+ if (graph_node_edges_add(graph))
+ goto graph_cleanup;
+
+ /* Update adjacency list of all nodes in the graph */
+ if (graph_adjacency_list_update(graph))
+ goto graph_cleanup;
+
+ /* Make sure at least a source node present in the graph */
+ if (!graph_src_nodes_count(graph))
+ goto graph_cleanup;
+
+ /* Make sure no node is pointing to source node */
+ if (graph_node_has_edge_to_src_node(graph))
+ goto graph_cleanup;
+
+ /* Don't allow node has loop to self */
+ if (graph_node_has_loop_edge(graph))
+ goto graph_cleanup;
+
+ /* Do BFS from src nodes on the graph to find isolated nodes */
+ if (graph_has_isolated_node(graph))
+ goto graph_cleanup;
+
+ /* Initialize graph object */
+ graph->socket = prm->socket_id;
+ graph->src_node_count = graph_src_nodes_count(graph);
+ graph->node_count = graph_nodes_count(graph);
+ graph->id = graph_id;
+
+ /* Allocate the Graph fast path memory and populate the data */
+ if (graph_fp_mem_create(graph))
+ goto graph_cleanup;
+
+ /* Call init() of the all the nodes in the graph */
+ if (graph_node_init(graph))
+ goto graph_mem_destroy;
+
+ /* All good, Lets add the graph to the list */
+ graph_id++;
+ STAILQ_INSERT_TAIL(&graph_list, graph, next);
+
+ graph_spinlock_unlock();
+ return graph->id;
+
+graph_mem_destroy:
+ graph_fp_mem_destroy(graph);
+graph_cleanup:
+ graph_cleanup(graph);
+free:
+ free(graph);
+fail:
+ graph_spinlock_unlock();
+ return RTE_GRAPH_ID_INVALID;
+}
+
+rte_graph_t
+rte_graph_destroy(const char *graph_name)
+{
+ rte_graph_t rc = RTE_GRAPH_ID_INVALID;
+ struct graph *graph, *tmp;
+ const char *name;
+
+ graph_spinlock_lock();
+
+ graph = STAILQ_FIRST(&graph_list);
+ while (graph != NULL) {
+ tmp = STAILQ_NEXT(graph, next);
+ name = graph->name;
+ if (strncmp(name, graph_name, RTE_GRAPH_NAMESIZE) == 0) {
+ /* Call fini() of the all the nodes in the graph */
+ graph_node_fini(graph);
+ /* Destroy graph fast path memory */
+ rc = graph_fp_mem_destroy(graph);
+ if (rc)
+ SET_ERR_JMP(rc, done, "MZ %s free failed",
+ name);
+
+ graph_cleanup(graph);
+ STAILQ_REMOVE(&graph_list, graph, graph, next);
+ rc = graph->id;
+ free(graph);
+ graph_id--;
+ goto done;
+ }
+ graph = tmp;
+ }
+done:
+ graph_spinlock_unlock();
+ return rc;
+}
+
void __rte_noinline
__rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
{
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index a9fe1b610..dcbd78c02 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -4,6 +4,8 @@ EXPERIMENTAL {
__rte_node_register;
__rte_node_stream_alloc;
+ rte_graph_create;
+ rte_graph_destroy;
rte_node_clone;
rte_node_dump;
rte_node_edge_count;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 08/26] graph: implement graph operation APIs
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (6 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 07/26] graph: implement create and destroy APIs jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 09/26] graph: implement Graphviz export jerinj
` (18 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K, Anatoly Burakov
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding support for graph specific API implementation like
Graph lookup to get graph object, retrieving graph ID
From name and graph name from ID.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph.c | 131 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 8 ++
2 files changed, 139 insertions(+)
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index d060ffe70..72a82c2a8 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -211,6 +211,54 @@ graph_node_fini(struct graph *graph)
graph_node->node->name));
}
+static struct rte_graph *
+graph_mem_fixup_node_ctx(struct rte_graph *graph)
+{
+ struct rte_node *node;
+ struct node *node_db;
+ rte_graph_off_t off;
+ rte_node_t count;
+ const char *name;
+
+ rte_graph_foreach_node(count, off, graph, node) {
+ if (node->parent_id == RTE_NODE_ID_INVALID) /* Static node */
+ name = node->name;
+ else /* Cloned node */
+ name = node->parent;
+
+ node_db = node_from_name(name);
+ if (node_db == NULL)
+ SET_ERR_JMP(ENOLINK, fail, "Node %s not found", name);
+ node->process = node_db->process;
+ }
+
+ return graph;
+fail:
+ return NULL;
+}
+
+static struct rte_graph *
+graph_mem_fixup_secondray(struct rte_graph *graph)
+{
+ if (graph == NULL || rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return graph;
+
+ return graph_mem_fixup_node_ctx(graph);
+}
+
+struct rte_graph *
+rte_graph_lookup(const char *name)
+{
+ const struct rte_memzone *mz;
+ struct rte_graph *rc = NULL;
+
+ mz = rte_memzone_lookup(name);
+ if (mz)
+ rc = mz->addr;
+
+ return graph_mem_fixup_secondray(rc);
+}
+
rte_graph_t
rte_graph_create(const char *name, struct rte_graph_param *prm)
{
@@ -342,6 +390,76 @@ rte_graph_destroy(const char *graph_name)
return rc;
}
+rte_graph_t
+rte_graph_from_name(const char *name)
+{
+ struct graph *graph;
+
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (strncmp(graph->name, name, RTE_GRAPH_NAMESIZE) == 0)
+ return graph->id;
+
+ return RTE_GRAPH_ID_INVALID;
+}
+
+char *
+rte_graph_id_to_name(rte_graph_t id)
+{
+ struct graph *graph;
+
+ GRAPH_ID_CHECK(id);
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (graph->id == id)
+ return graph->name;
+
+fail:
+ return NULL;
+}
+
+struct rte_node *
+rte_graph_node_get(rte_graph_t gid, uint32_t nid)
+{
+ struct rte_node *node;
+ struct graph *graph;
+ rte_graph_off_t off;
+ rte_node_t count;
+
+ GRAPH_ID_CHECK(gid);
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (graph->id == gid) {
+ rte_graph_foreach_node(count, off, graph->graph,
+ node) {
+ if (node->id == nid)
+ return node;
+ }
+ break;
+ }
+fail:
+ return NULL;
+}
+
+struct rte_node *
+rte_graph_node_get_by_name(const char *graph_name, const char *node_name)
+{
+ struct rte_node *node;
+ struct graph *graph;
+ rte_graph_off_t off;
+ rte_node_t count;
+
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (!strncmp(graph->name, graph_name, RTE_GRAPH_NAMESIZE)) {
+ rte_graph_foreach_node(count, off, graph->graph,
+ node) {
+ if (!strncmp(node->name, node_name,
+ RTE_NODE_NAMESIZE))
+ return node;
+ }
+ break;
+ }
+
+ return NULL;
+}
+
void __rte_noinline
__rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
{
@@ -356,3 +474,16 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
node->size = size;
node->realloc_count++;
}
+
+rte_graph_t
+rte_graph_max_count(void)
+{
+ return graph_id;
+}
+
+RTE_INIT(rte_graph_init_log)
+{
+ rte_graph_logtype = rte_log_register("lib.graph");
+ if (rte_graph_logtype >= 0)
+ rte_log_set_level(rte_graph_logtype, RTE_LOG_INFO);
+}
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index dcbd78c02..5a2b13293 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -6,6 +6,14 @@ EXPERIMENTAL {
rte_graph_create;
rte_graph_destroy;
+ rte_graph_from_name;
+ rte_graph_id_to_name;
+ rte_graph_lookup;
+ rte_graph_list_dump;
+ rte_graph_max_count;
+ rte_graph_node_get;
+ rte_graph_node_get_by_name;
+
rte_node_clone;
rte_node_dump;
rte_node_edge_count;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 09/26] graph: implement Graphviz export
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (7 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 08/26] graph: implement graph operation APIs jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 10/26] graph: implement debug routines jerinj
` (17 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding API implementation support exporting the graph object to file.
This will export the graph to a file in Graphviz format.
It can be viewed in many viewers such as
https://dreampuf.github.io/GraphvizOnline/
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph.c | 54 ++++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 1 +
2 files changed, 55 insertions(+)
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index 72a82c2a8..ad58673be 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -475,6 +475,60 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
node->realloc_count++;
}
+static int
+graph_to_dot(FILE *f, struct graph *graph)
+{
+ const char *src_edge_color = " [color=blue]\n";
+ const char *edge_color = "\n";
+ struct graph_node *graph_node;
+ char *node_name;
+ rte_edge_t i;
+ int rc;
+
+ rc = fprintf(f, "Digraph %s {\n\trankdir=LR;\n", graph->name);
+ if (rc < 0)
+ goto end;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ node_name = graph_node->node->name;
+ for (i = 0; i < graph_node->node->nb_edges; i++) {
+ rc = fprintf(f, "\t\"%s\"->\"%s\"%s", node_name,
+ graph_node->adjacency_list[i]->node->name,
+ graph_node->node->flags & RTE_NODE_SOURCE_F
+ ? src_edge_color
+ : edge_color);
+ if (rc < 0)
+ goto end;
+ }
+ }
+ rc = fprintf(f, "}\n");
+ if (rc < 0)
+ goto end;
+
+ return 0;
+end:
+ rte_errno = EBADF;
+ return -rte_errno;
+}
+
+rte_graph_t
+rte_graph_export(const char *name, FILE *f)
+{
+ rte_graph_t rc = RTE_GRAPH_ID_INVALID;
+ struct graph *graph;
+
+ STAILQ_FOREACH(graph, &graph_list, next) {
+ if (strncmp(graph->name, name, RTE_GRAPH_NAMESIZE) == 0) {
+ rc = graph_to_dot(f, graph);
+ goto end;
+ }
+ }
+ rte_errno = ENOENT;
+end:
+ return rc;
+}
+
+
rte_graph_t
rte_graph_max_count(void)
{
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 5a2b13293..2797be044 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -6,6 +6,7 @@ EXPERIMENTAL {
rte_graph_create;
rte_graph_destroy;
+ rte_graph_export;
rte_graph_from_name;
rte_graph_id_to_name;
rte_graph_lookup;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 10/26] graph: implement debug routines
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (8 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 09/26] graph: implement Graphviz export jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 11/26] graph: implement stats support jerinj
` (16 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding implementation for graph specific API to dump the
Graph information to a file. This API will dump detailed internal
info about node objects and graph objects.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph.c | 31 ++++++++++++++
lib/librte_graph/graph_debug.c | 59 ++++++++++++++++++++++++++
lib/librte_graph/graph_private.h | 13 ++++++
lib/librte_graph/rte_graph_version.map | 2 +
4 files changed, 105 insertions(+)
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index ad58673be..cc1e523d9 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -528,6 +528,37 @@ rte_graph_export(const char *name, FILE *f)
return rc;
}
+static void
+graph_scan_dump(FILE *f, rte_graph_t id, bool all)
+{
+ struct graph *graph;
+
+ RTE_VERIFY(f);
+ GRAPH_ID_CHECK(id);
+
+ STAILQ_FOREACH(graph, &graph_list, next) {
+ if (all == true) {
+ graph_dump(f, graph);
+ } else if (graph->id == id) {
+ graph_dump(f, graph);
+ return;
+ }
+ }
+fail:
+ return;
+}
+
+void
+rte_graph_dump(FILE *f, rte_graph_t id)
+{
+ graph_scan_dump(f, id, false);
+}
+
+void
+rte_graph_list_dump(FILE *f)
+{
+ graph_scan_dump(f, 0, true);
+}
rte_graph_t
rte_graph_max_count(void)
diff --git a/lib/librte_graph/graph_debug.c b/lib/librte_graph/graph_debug.c
index 75238e7ca..f8aea16ac 100644
--- a/lib/librte_graph/graph_debug.c
+++ b/lib/librte_graph/graph_debug.c
@@ -7,6 +7,26 @@
#include "graph_private.h"
+void
+graph_dump(FILE *f, struct graph *g)
+{
+ struct graph_node *graph_node;
+ rte_edge_t i = 0;
+
+ fprintf(f, "graph <%s>\n", g->name);
+ fprintf(f, " id=%" PRIu32 "\n", g->id);
+ fprintf(f, " cir_start=%" PRIu32 "\n", g->cir_start);
+ fprintf(f, " cir_mask=%" PRIu32 "\n", g->cir_mask);
+ fprintf(f, " addr=%p\n", g);
+ fprintf(f, " graph=%p\n", g->graph);
+ fprintf(f, " mem_sz=%zu\n", g->mem_sz);
+ fprintf(f, " node_count=%" PRIu32 "\n", g->node_count);
+ fprintf(f, " src_node_count=%" PRIu32 "\n", g->src_node_count);
+
+ STAILQ_FOREACH(graph_node, &g->node_list, next)
+ fprintf(f, " node[%d] <%s>\n", i++, graph_node->node->name);
+}
+
void
node_dump(FILE *f, struct node *n)
{
@@ -23,3 +43,42 @@ node_dump(FILE *f, struct node *n)
fprintf(f, " edge[%d] <%s>\n", i, n->next_nodes[i]);
}
+void
+rte_graph_obj_dump(FILE *f, struct rte_graph *g, bool all)
+{
+ rte_node_t count;
+ rte_graph_off_t off;
+ struct rte_node *n;
+ rte_edge_t i;
+
+ fprintf(f, "graph <%s> @ %p\n", g->name, g);
+ fprintf(f, " id=%" PRIu32 "\n", g->id);
+ fprintf(f, " head=%" PRId32 "\n", (int32_t)g->head);
+ fprintf(f, " tail=%" PRId32 "\n", (int32_t)g->tail);
+ fprintf(f, " cir_mask=0x%" PRIx32 "\n", g->cir_mask);
+ fprintf(f, " nb_nodes=%" PRId32 "\n", g->nb_nodes);
+ fprintf(f, " socket=%d\n", g->socket);
+ fprintf(f, " fence=0x%" PRIx64 "\n", g->fence);
+ fprintf(f, " nodes_start=0x%" PRIx32 "\n", g->nodes_start);
+ fprintf(f, " cir_start=%p\n", g->cir_start);
+
+ rte_graph_foreach_node(count, off, g, n) {
+ if (!all && n->idx == 0)
+ continue;
+ fprintf(f, " node[%d] <%s>\n", count, n->name);
+ fprintf(f, " fence=0x%" PRIx64 "\n", n->fence);
+ fprintf(f, " objs=%p\n", n->objs);
+ fprintf(f, " process=%p\n", n->process);
+ fprintf(f, " id=0x%" PRIx32 "\n", n->id);
+ fprintf(f, " offset=0x%" PRIx32 "\n", n->off);
+ fprintf(f, " nb_edges=%" PRId32 "\n", n->nb_edges);
+ fprintf(f, " realloc_count=%d\n", n->realloc_count);
+ fprintf(f, " size=%d\n", n->size);
+ fprintf(f, " idx=%d\n", n->idx);
+ fprintf(f, " total_objs=%" PRId64 "\n", n->total_objs);
+ fprintf(f, " total_calls=%" PRId64 "\n", n->total_calls);
+ for (i = 0; i < n->nb_edges; i++)
+ fprintf(f, " edge[%d] <%s>\n", i,
+ n->nodes[i]->name);
+ }
+}
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index 051fad53a..397247797 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -318,6 +318,19 @@ struct rte_node *graph_node_id_to_ptr(const struct rte_graph *graph,
struct rte_node *graph_node_name_to_ptr(const struct rte_graph *graph,
const char *node_name);
+/* Debug functions */
+/**
+ * @internal
+ *
+ * Dump internal graph object data.
+ *
+ * @param f
+ * FILE pointer to dump the data.
+ * @param g
+ * Pointer to the internal graph object.
+ */
+void graph_dump(FILE *f, struct graph *g);
+
/**
* @internal
*
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 2797be044..851f4772e 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -6,6 +6,7 @@ EXPERIMENTAL {
rte_graph_create;
rte_graph_destroy;
+ rte_graph_dump;
rte_graph_export;
rte_graph_from_name;
rte_graph_id_to_name;
@@ -14,6 +15,7 @@ EXPERIMENTAL {
rte_graph_max_count;
rte_graph_node_get;
rte_graph_node_get_by_name;
+ rte_graph_obj_dump;
rte_node_clone;
rte_node_dump;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 11/26] graph: implement stats support
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (9 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 10/26] graph: implement debug routines jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 12/26] graph: implement fastpath API routines jerinj
` (15 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding implementation for graph stats collection API. This API will
create a cluster for a specified node pattern and aggregate the node
runtime stats.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
lib/librte_graph/Makefile | 1 +
lib/librte_graph/graph_stats.c | 406 +++++++++++++++++++++++++
lib/librte_graph/meson.build | 2 +-
lib/librte_graph/rte_graph_version.map | 5 +
4 files changed, 413 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_graph/graph_stats.c
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 7bfd7d51f..967c8d9bc 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -18,6 +18,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_stats.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_populate.c
# install header files
diff --git a/lib/librte_graph/graph_stats.c b/lib/librte_graph/graph_stats.c
new file mode 100644
index 000000000..125e08d73
--- /dev/null
+++ b/lib/librte_graph/graph_stats.c
@@ -0,0 +1,406 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <fnmatch.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_malloc.h>
+
+#include "graph_private.h"
+
+/* Capture all graphs of cluster */
+struct cluster {
+ rte_graph_t nb_graphs;
+ rte_graph_t size;
+
+ struct graph **graphs;
+};
+
+/* Capture same node ID across cluster */
+struct cluster_node {
+ struct rte_graph_cluster_node_stats stat;
+ rte_node_t nb_nodes;
+
+ struct rte_node *nodes[];
+};
+
+struct rte_graph_cluster_stats {
+ /* Header */
+ rte_graph_cluster_stats_cb_t fn;
+ uint32_t cluster_node_size; /* Size of struct cluster_node */
+ rte_node_t max_nodes;
+ int socket_id;
+ void *cookie;
+ size_t sz;
+
+ struct cluster_node clusters[];
+} __rte_cache_aligned;
+
+#define boarder() \
+ fprintf(f, "+-------------------------------+---------------+--------" \
+ "-------+---------------+---------------+---------------+-" \
+ "----------+\n")
+
+static inline void
+print_banner(FILE *f)
+{
+ boarder();
+ fprintf(f, "%-32s%-16s%-16s%-16s%-16s%-16s%-16s\n", "|Node", "|calls",
+ "|objs", "|realloc_count", "|objs/call", "|objs/sec(10E6)",
+ "|cycles/call|");
+ boarder();
+}
+
+static inline void
+print_node(FILE *f, const struct rte_graph_cluster_node_stats *stat)
+{
+ double objs_per_call, objs_per_sec, cycles_per_call, ts_per_hz;
+ const uint64_t prev_calls = stat->prev_calls;
+ const uint64_t prev_objs = stat->prev_objs;
+ const uint64_t cycles = stat->cycles;
+ const uint64_t calls = stat->calls;
+ const uint64_t objs = stat->objs;
+ uint64_t call_delta;
+
+ call_delta = calls - prev_calls;
+ objs_per_call =
+ call_delta ? (double)((objs - prev_objs) / call_delta) : 0;
+ cycles_per_call =
+ call_delta ? (double)((cycles - stat->prev_cycles) / call_delta)
+ : 0;
+ ts_per_hz = (double)((stat->ts - stat->prev_ts) / stat->hz);
+ objs_per_sec = ts_per_hz ? (objs - prev_objs) / ts_per_hz : 0;
+ objs_per_sec /= 1000000;
+
+ fprintf(f,
+ "|%-31s|%-15" PRIu64 "|%-15" PRIu64 "|%-15" PRIu64
+ "|%-15.3f|%-15.6f|%-11.4f|\n",
+ stat->name, calls, objs, stat->realloc_count, objs_per_call,
+ objs_per_sec, cycles_per_call);
+}
+
+static int
+graph_cluster_stats_cb(bool is_first, bool is_last, void *cookie,
+ const struct rte_graph_cluster_node_stats *stat)
+{
+ FILE *f = cookie;
+
+ if (unlikely(is_first))
+ print_banner(f);
+ if (stat->objs)
+ print_node(f, stat);
+ if (unlikely(is_last))
+ boarder();
+
+ return 0;
+};
+
+static struct rte_graph_cluster_stats *
+stats_mem_init(struct cluster *cluster,
+ const struct rte_graph_cluster_stats_param *prm)
+{
+ size_t sz = sizeof(struct rte_graph_cluster_stats);
+ struct rte_graph_cluster_stats *stats;
+ rte_graph_cluster_stats_cb_t fn;
+ int socket_id = prm->socket_id;
+ uint32_t cluster_node_size;
+
+ /* Fix up callback */
+ fn = prm->fn;
+ if (fn == NULL)
+ fn = graph_cluster_stats_cb;
+
+ cluster_node_size = sizeof(struct cluster_node);
+ /* For a given cluster, max nodes will be the max number of graphs */
+ cluster_node_size += cluster->nb_graphs * sizeof(struct rte_node *);
+ cluster_node_size = RTE_ALIGN(cluster_node_size, RTE_CACHE_LINE_SIZE);
+
+ stats = realloc(NULL, sz);
+ memset(stats, 0, sz);
+ if (stats) {
+ stats->fn = fn;
+ stats->cluster_node_size = cluster_node_size;
+ stats->max_nodes = 0;
+ stats->socket_id = socket_id;
+ stats->cookie = prm->cookie;
+ stats->sz = sz;
+ }
+
+ return stats;
+}
+
+static int
+stats_mem_populate(struct rte_graph_cluster_stats **stats_in,
+ struct rte_graph *graph, struct graph_node *graph_node)
+{
+ struct rte_graph_cluster_stats *stats = *stats_in;
+ rte_node_t id = graph_node->node->id;
+ struct cluster_node *cluster;
+ struct rte_node *node;
+ rte_node_t count;
+
+ cluster = stats->clusters;
+
+ /* Iterate over cluster node array to find node ID match */
+ for (count = 0; count < stats->max_nodes; count++) {
+ /* Found an existing node in the reel */
+ if (cluster->stat.id == id) {
+ node = graph_node_id_to_ptr(graph, id);
+ if (node == NULL)
+ SET_ERR_JMP(
+ ENOENT, err,
+ "Failed to find node %s in graph %s",
+ graph_node->node->name, graph->name);
+
+ cluster->nodes[cluster->nb_nodes++] = node;
+ return 0;
+ }
+ cluster = RTE_PTR_ADD(cluster, stats->cluster_node_size);
+ }
+
+ /* Hey, it is a new node, allocate space for it in the reel */
+ stats = realloc(stats, stats->sz + stats->cluster_node_size);
+ if (stats == NULL)
+ SET_ERR_JMP(ENOMEM, err, "Realloc failed");
+
+ /* Clear the new struct cluster_node area */
+ cluster = RTE_PTR_ADD(stats, stats->sz),
+ memset(cluster, 0, stats->cluster_node_size);
+ memcpy(cluster->stat.name, graph_node->node->name, RTE_NODE_NAMESIZE);
+ cluster->stat.id = graph_node->node->id;
+ cluster->stat.hz = rte_get_timer_hz();
+ node = graph_node_id_to_ptr(graph, id);
+ if (node == NULL)
+ SET_ERR_JMP(ENOENT, err, "Failed to find node %s in graph %s",
+ graph_node->node->name, graph->name);
+ cluster->nodes[cluster->nb_nodes++] = node;
+
+ stats->sz += stats->cluster_node_size;
+ stats->max_nodes++;
+ *stats_in = stats;
+
+ return 0;
+err:
+ return -rte_errno;
+}
+
+static void
+stats_mem_fini(struct rte_graph_cluster_stats *stats)
+{
+ free(stats);
+}
+
+static void
+cluster_init(struct cluster *cluster)
+{
+ memset(cluster, 0, sizeof(*cluster));
+}
+
+static int
+cluster_add(struct cluster *cluster, struct graph *graph)
+{
+ rte_graph_t count;
+ size_t sz;
+
+ /* Skip the if graph is already added to cluster */
+ for (count = 0; count < cluster->nb_graphs; count++)
+ if (cluster->graphs[count] == graph)
+ return 0;
+
+ /* Expand the cluster if required to store graph objects */
+ if (cluster->nb_graphs + 1 > cluster->size) {
+ cluster->size = RTE_MAX(1, cluster->size * 2);
+ sz = sizeof(struct graph *) * cluster->size;
+ cluster->graphs = realloc(cluster->graphs, sz);
+ if (cluster->graphs == NULL)
+ SET_ERR_JMP(ENOMEM, free, "Failed to realloc");
+ }
+
+ /* Add graph to cluster */
+ cluster->graphs[cluster->nb_graphs++] = graph;
+ return 0;
+
+free:
+ return -rte_errno;
+}
+
+static void
+cluster_fini(struct cluster *cluster)
+{
+ if (cluster->graphs)
+ free(cluster->graphs);
+}
+
+static int
+expand_pattern_to_cluster(struct cluster *cluster, const char *pattern)
+{
+ struct graph_head *graph_head = graph_list_head_get();
+ struct graph *graph;
+ bool found = false;
+
+ /* Check for pattern match */
+ STAILQ_FOREACH(graph, graph_head, next) {
+ if (fnmatch(pattern, graph->name, 0) == 0) {
+ if (cluster_add(cluster, graph))
+ goto fail;
+ found = true;
+ }
+ }
+ if (found == false)
+ SET_ERR_JMP(EFAULT, fail, "Pattern %s graph not found",
+ pattern);
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+struct rte_graph_cluster_stats *
+rte_graph_cluster_stats_create(const struct rte_graph_cluster_stats_param *prm)
+{
+ struct rte_graph_cluster_stats *stats, *rc = NULL;
+ struct graph_node *graph_node;
+ struct cluster cluster;
+ struct graph *graph;
+ const char *pattern;
+ rte_graph_t i;
+
+ /* Sanity checks */
+ if (!rte_graph_has_stats_feature())
+ SET_ERR_JMP(EINVAL, fail, "Stats feature is not enabled");
+
+ if (prm == NULL)
+ SET_ERR_JMP(EINVAL, fail, "Invalid param");
+
+ if (prm->graph_patterns == NULL || prm->nb_graph_patterns == 0)
+ SET_ERR_JMP(EINVAL, fail, "Invalid graph param");
+
+ cluster_init(&cluster);
+
+ graph_spinlock_lock();
+ /* Expand graph pattern and add the graph to the cluster */
+ for (i = 0; i < prm->nb_graph_patterns; i++) {
+ pattern = prm->graph_patterns[i];
+ if (expand_pattern_to_cluster(&cluster, pattern))
+ goto bad_pattern;
+ }
+
+ /* Alloc the stats memory */
+ stats = stats_mem_init(&cluster, prm);
+ if (stats == NULL)
+ SET_ERR_JMP(ENOMEM, bad_pattern, "Failed alloc stats memory");
+
+ /* Iterate over M(Graph) x N (Nodes in graph) */
+ for (i = 0; i < cluster.nb_graphs; i++) {
+ graph = cluster.graphs[i];
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ struct rte_graph *graph_fp = graph->graph;
+ if (stats_mem_populate(&stats, graph_fp, graph_node))
+ goto realloc_fail;
+ }
+ }
+
+ /* Finally copy to hugepage memory to avoid pressure on rte_realloc */
+ rc = rte_malloc_socket(NULL, stats->sz, 0, stats->socket_id);
+ if (rc)
+ rte_memcpy(rc, stats, stats->sz);
+ else
+ SET_ERR_JMP(ENOMEM, realloc_fail, "rte_malloc failed");
+
+realloc_fail:
+ stats_mem_fini(stats);
+bad_pattern:
+ graph_spinlock_unlock();
+ cluster_fini(&cluster);
+fail:
+ return rc;
+}
+
+void
+rte_graph_cluster_stats_destroy(struct rte_graph_cluster_stats *stat)
+{
+ return rte_free(stat);
+}
+
+static inline void
+cluster_node_arregate_stats(struct cluster_node *cluster)
+{
+ uint64_t calls = 0, cycles = 0, objs = 0, realloc_count = 0;
+ struct rte_graph_cluster_node_stats *stat = &cluster->stat;
+ struct rte_node *node;
+ rte_node_t count;
+
+ for (count = 0; count < cluster->nb_nodes; count++) {
+ node = cluster->nodes[count];
+
+ calls += node->total_calls;
+ objs += node->total_objs;
+ cycles += node->total_cycles;
+ realloc_count += node->realloc_count;
+ }
+
+ stat->calls = calls;
+ stat->objs = objs;
+ stat->cycles = cycles;
+ stat->ts = rte_get_timer_cycles();
+ stat->realloc_count = realloc_count;
+}
+
+static inline void
+cluster_node_store_prev_stats(struct cluster_node *cluster)
+{
+ struct rte_graph_cluster_node_stats *stat = &cluster->stat;
+
+ stat->prev_ts = stat->ts;
+ stat->prev_calls = stat->calls;
+ stat->prev_objs = stat->objs;
+ stat->prev_cycles = stat->cycles;
+}
+
+void
+rte_graph_cluster_stats_get(struct rte_graph_cluster_stats *stat, bool skip_cb)
+{
+ struct cluster_node *cluster;
+ rte_node_t count;
+ int rc = 0;
+
+ cluster = stat->clusters;
+
+ for (count = 0; count < stat->max_nodes; count++) {
+ cluster_node_arregate_stats(cluster);
+ if (!skip_cb)
+ rc = stat->fn(!count, (count == stat->max_nodes - 1),
+ stat->cookie, &cluster->stat);
+ cluster_node_store_prev_stats(cluster);
+ if (rc)
+ break;
+ cluster = RTE_PTR_ADD(cluster, stat->cluster_node_size);
+ }
+}
+
+void
+rte_graph_cluster_stats_reset(struct rte_graph_cluster_stats *stat)
+{
+ struct cluster_node *cluster;
+ rte_node_t count;
+
+ cluster = stat->clusters;
+
+ for (count = 0; count < stat->max_nodes; count++) {
+ struct rte_graph_cluster_node_stats *node = &cluster->stat;
+
+ node->ts = 0;
+ node->calls = 0;
+ node->objs = 0;
+ node->cycles = 0;
+ node->prev_ts = 0;
+ node->prev_calls = 0;
+ node->prev_objs = 0;
+ node->prev_cycles = 0;
+ node->realloc_count = 0;
+ cluster = RTE_PTR_ADD(cluster, stat->cluster_node_size);
+ }
+}
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index fb203a5e2..929a17f84 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,7 +4,7 @@
name = 'graph'
-sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_populate.c')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_stats.c', 'graph_populate.c')
headers = files('rte_graph.h', 'rte_graph_worker.h')
allow_experimental_apis = true
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 851f4772e..adf55d406 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -17,6 +17,11 @@ EXPERIMENTAL {
rte_graph_node_get_by_name;
rte_graph_obj_dump;
+ rte_graph_cluster_stats_create;
+ rte_graph_cluster_stats_destroy;
+ rte_graph_cluster_stats_get;
+ rte_graph_cluster_stats_reset;
+
rte_node_clone;
rte_node_dump;
rte_node_edge_count;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 12/26] graph: implement fastpath API routines
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (10 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 11/26] graph: implement stats support jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 13/26] graph: add unit test case jerinj
` (14 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: John McNamara, Marko Kovacevic, Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding implementation for rte_graph_walk() API. This will perform a walk
on the circular buffer and call the process function of each node
and collect the stats if stats collection is enabled.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
doc/api/doxy-api-index.md | 1 +
lib/librte_graph/graph.c | 16 +
lib/librte_graph/rte_graph_version.map | 10 +
lib/librte_graph/rte_graph_worker.h | 434 +++++++++++++++++++++++++
4 files changed, 461 insertions(+)
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 5cc50f750..fd2ff64d7 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -160,6 +160,7 @@ The public API headers are grouped by topics:
[port_in_action] (@ref rte_port_in_action.h)
[table_action] (@ref rte_table_action.h)
* [graph] (@ref rte_graph.h):
+ [graph_worker] (@ref rte_graph_worker.h)
- **basic**:
[approx fraction] (@ref rte_approx.h),
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index cc1e523d9..78bc83c4e 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -475,6 +475,22 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
node->realloc_count++;
}
+void __rte_noinline
+__rte_node_stream_alloc_size(struct rte_graph *graph, struct rte_node *node,
+ uint16_t req_size)
+{
+ uint16_t size = node->size;
+
+ RTE_VERIFY(size != UINT16_MAX);
+ /* Allocate double amount of size to avoid immediate realloc */
+ size = RTE_MIN(UINT16_MAX, RTE_MAX(RTE_GRAPH_BURST_SIZE, req_size * 2));
+ node->objs = rte_realloc_socket(node->objs, size * sizeof(void *),
+ RTE_CACHE_LINE_SIZE, graph->socket);
+ RTE_VERIFY(node->objs);
+ node->size = size;
+ node->realloc_count++;
+}
+
static int
graph_to_dot(FILE *f, struct graph *graph)
{
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index adf55d406..13b838752 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -3,6 +3,7 @@ EXPERIMENTAL {
__rte_node_register;
__rte_node_stream_alloc;
+ __rte_node_stream_alloc_size;
rte_graph_create;
rte_graph_destroy;
@@ -16,6 +17,7 @@ EXPERIMENTAL {
rte_graph_node_get;
rte_graph_node_get_by_name;
rte_graph_obj_dump;
+ rte_graph_walk;
rte_graph_cluster_stats_create;
rte_graph_cluster_stats_destroy;
@@ -28,10 +30,18 @@ EXPERIMENTAL {
rte_node_edge_get;
rte_node_edge_shrink;
rte_node_edge_update;
+ rte_node_enqueue;
+ rte_node_enqueue_x1;
+ rte_node_enqueue_x2;
+ rte_node_enqueue_x4;
+ rte_node_enqueue_next;
rte_node_from_name;
rte_node_id_to_name;
rte_node_list_dump;
rte_node_max_count;
+ rte_node_next_stream_get;
+ rte_node_next_stream_put;
+ rte_node_next_stream_move;
local: *;
};
diff --git a/lib/librte_graph/rte_graph_worker.h b/lib/librte_graph/rte_graph_worker.h
index a7c780d4d..8e067e673 100644
--- a/lib/librte_graph/rte_graph_worker.h
+++ b/lib/librte_graph/rte_graph_worker.h
@@ -100,6 +100,440 @@ struct rte_node {
__rte_experimental
void __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node);
+/**
+ * @internal
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Allocate a stream with requested number of objects.
+ *
+ * If stream already exists then re-allocate it to a larger size.
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ * @param req_size
+ * Number of objects to be allocated.
+ */
+__rte_experimental
+void __rte_node_stream_alloc_size(struct rte_graph *graph,
+ struct rte_node *node, uint16_t req_size);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Perform graph walk on the circular buffer and invoke the process function
+ * of the nodes and collect the stats.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup function.
+ *
+ * @see rte_graph_lookup()
+ */
+__rte_experimental
+static inline void
+rte_graph_walk(struct rte_graph *graph)
+{
+ const rte_graph_off_t *cir_start = graph->cir_start;
+ const rte_node_t mask = graph->cir_mask;
+ uint32_t head = graph->head;
+ struct rte_node *node;
+ uint64_t start;
+ uint16_t rc;
+ void **objs;
+
+ /*
+ * Walk on the source node(s) ((cir_start - head) -> cir_start) and then
+ * on the pending streams (cir_start -> (cir_start + mask) -> cir_start)
+ * in a circular buffer fashion.
+ *
+ * +-----+ <= cir_start - head [number of source nodes]
+ * | |
+ * | ... | <= source nodes
+ * | |
+ * +-----+ <= cir_start [head = 0] [tail = 0]
+ * | |
+ * | ... | <= pending streams
+ * | |
+ * +-----+ <= cir_start + mask
+ */
+ while (likely(head != graph->tail)) {
+ node = RTE_PTR_ADD(graph, cir_start[(int32_t)head++]);
+ RTE_ASSERT(node->fence == RTE_GRAPH_FENCE);
+ objs = node->objs;
+ rte_prefetch0(objs);
+
+ if (rte_graph_has_stats_feature()) {
+ start = rte_rdtsc();
+ rc = node->process(graph, node, objs, node->idx);
+ node->total_cycles += rte_rdtsc() - start;
+ node->total_calls++;
+ node->total_objs += rc;
+ } else {
+ node->process(graph, node, objs, node->idx);
+ }
+ node->idx = 0;
+ head = likely((int32_t)head > 0) ? head & mask : head;
+ }
+ graph->tail = 0;
+}
+
+/* Fast path helper functions */
+
+/**
+ * @internal
+ *
+ * Enqueue a given node to the tail of the graph reel.
+ *
+ * @param graph
+ * Pointer Graph object.
+ * @param node
+ * Pointer to node object to be enqueued.
+ */
+static __rte_always_inline void
+__rte_node_enqueue_tail_update(struct rte_graph *graph, struct rte_node *node)
+{
+ uint32_t tail;
+
+ tail = graph->tail;
+ graph->cir_start[tail++] = node->off;
+ graph->tail = tail & graph->cir_mask;
+}
+
+/**
+ * @internal
+ *
+ * Enqueue sequence prologue function.
+ *
+ * Updates the node to tail of graph reel and resizes the number of objects
+ * available in the stream as needed.
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ * @param idx
+ * Index at which the object enqueue starts from.
+ * @param space
+ * Space required for the object enqueue.
+ */
+static __rte_always_inline void
+__rte_node_enqueue_prologue(struct rte_graph *graph, struct rte_node *node,
+ const uint16_t idx, const uint16_t space)
+{
+
+ /* Add to the pending stream list if the node is new */
+ if (idx == 0)
+ __rte_node_enqueue_tail_update(graph, node);
+
+ if (unlikely(node->size < (idx + space)))
+ __rte_node_stream_alloc(graph, node);
+}
+
+/**
+ * @internal
+ *
+ * Get the node pointer from current node edge id.
+ *
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Edge id of the required node.
+ *
+ * @return
+ * Pointer to the node denoted by the edge id.
+ */
+static __rte_always_inline struct rte_node *
+__rte_node_next_node_get(struct rte_node *node, rte_edge_t next)
+{
+ RTE_ASSERT(next < node->nb_edges);
+ RTE_ASSERT(node->fence == RTE_GRAPH_FENCE);
+ node = node->nodes[next];
+ RTE_ASSERT(node->fence == RTE_GRAPH_FENCE);
+
+ return node;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue the objs to next node for further processing and set
+ * the next node to pending state in the circular buffer.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to enqueue objs.
+ * @param objs
+ * Objs to enqueue.
+ * @param nb_objs
+ * Number of objs to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, void **objs, uint16_t nb_objs)
+{
+ node = __rte_node_next_node_get(node, next);
+ const uint16_t idx = node->idx;
+
+ __rte_node_enqueue_prologue(graph, node, idx, nb_objs);
+
+ rte_memcpy(&node->objs[idx], objs, nb_objs * sizeof(void *));
+ node->idx = idx + nb_objs;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue only one obj to next node for further processing and
+ * set the next node to pending state in the circular buffer.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to enqueue objs.
+ * @param obj
+ * Obj to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue_x1(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, void *obj)
+{
+ node = __rte_node_next_node_get(node, next);
+ uint16_t idx = node->idx;
+
+ __rte_node_enqueue_prologue(graph, node, idx, 1);
+
+ node->objs[idx++] = obj;
+ node->idx = idx;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue only two objs to next node for further processing and
+ * set the next node to pending state in the circular buffer.
+ * Same as rte_node_enqueue_x1 but enqueue two objs.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to enqueue objs.
+ * @param obj0
+ * Obj to enqueue.
+ * @param obj1
+ * Obj to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue_x2(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, void *obj0, void *obj1)
+{
+ node = __rte_node_next_node_get(node, next);
+ uint16_t idx = node->idx;
+
+ __rte_node_enqueue_prologue(graph, node, idx, 2);
+
+ node->objs[idx++] = obj0;
+ node->objs[idx++] = obj1;
+ node->idx = idx;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue only four objs to next node for further processing and
+ * set the next node to pending state in the circular buffer.
+ * Same as rte_node_enqueue_x1 but enqueue four objs.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to enqueue objs.
+ * @param obj0
+ * 1st obj to enqueue.
+ * @param obj1
+ * 2nd obj to enqueue.
+ * @param obj2
+ * 3rd obj to enqueue.
+ * @param obj3
+ * 4th obj to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue_x4(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, void *obj0, void *obj1, void *obj2,
+ void *obj3)
+{
+ node = __rte_node_next_node_get(node, next);
+ uint16_t idx = node->idx;
+
+ __rte_node_enqueue_prologue(graph, node, idx, 4);
+
+ node->objs[idx++] = obj0;
+ node->objs[idx++] = obj1;
+ node->objs[idx++] = obj2;
+ node->objs[idx++] = obj3;
+ node->idx = idx;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue objs to multiple next nodes for further processing and
+ * set the next nodes to pending state in the circular buffer.
+ * objs[i] will be enqueued to nexts[i].
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param nexts
+ * List of relative next node indices to enqueue objs.
+ * @param objs
+ * List of objs to enqueue.
+ * @param nb_objs
+ * Number of objs to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue_next(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t *nexts, void **objs, uint16_t nb_objs)
+{
+ uint16_t i;
+
+ for (i = 0; i < nb_objs; i++)
+ rte_node_enqueue_x1(graph, node, nexts[i], objs[i]);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get the stream of next node to enqueue the objs.
+ * Once done with the updating the objs, needs to call
+ * rte_node_next_stream_put to put the next node to pending state.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to get stream.
+ * @param nb_objs
+ * Requested free size of the next stream.
+ *
+ * @return
+ * Valid next stream on success.
+ *
+ * @see rte_node_next_stream_put().
+ */
+__rte_experimental
+static inline void **
+rte_node_next_stream_get(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, uint16_t nb_objs)
+{
+ node = __rte_node_next_node_get(node, next);
+ const uint16_t idx = node->idx;
+ uint16_t free_space = node->size - idx;
+
+ if (unlikely(free_space < nb_objs))
+ __rte_node_stream_alloc_size(graph, node, nb_objs);
+
+ return &node->objs[idx];
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Put the next stream to pending state in the circular buffer
+ * for further processing. Should be invoked followed by
+ * rte_node_next_stream_get().
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index..
+ * @param idx
+ * Number of objs updated in the stream after getting the stream using
+ * rte_node_next_stream_get.
+ *
+ * @see rte_node_next_stream_get().
+ */
+__rte_experimental
+static inline void
+rte_node_next_stream_put(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, uint16_t idx)
+{
+ if (unlikely(!idx))
+ return;
+
+ node = __rte_node_next_node_get(node, next);
+ if (node->idx == 0)
+ __rte_node_enqueue_tail_update(graph, node);
+
+ node->idx += idx;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Home run scenario, Enqueue all the objs of current node to next
+ * node in optimized way by swapping the streams of both nodes.
+ * Performs good when next node is already not in pending state.
+ * If next node is already in pending state then normal enqueue
+ * will be used.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param src
+ * Current node pointer.
+ * @param next
+ * Relative next node index.
+ */
+__rte_experimental
+static inline void
+rte_node_next_stream_move(struct rte_graph *graph, struct rte_node *src,
+ rte_edge_t next)
+{
+ struct rte_node *dst = __rte_node_next_node_get(src, next);
+
+ /* Let swap the pointers if dst don't have valid objs */
+ if (likely(dst->idx == 0)) {
+ void **dobjs = dst->objs;
+ uint16_t dsz = dst->size;
+ dst->objs = src->objs;
+ dst->size = src->size;
+ src->objs = dobjs;
+ src->size = dsz;
+ dst->idx = src->idx;
+ __rte_node_enqueue_tail_update(graph, dst);
+ } else { /* Move the objects from src node to dst node */
+ rte_node_enqueue(graph, src, next, src->objs, src->idx);
+ }
+}
+
#ifdef __cplusplus
}
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 13/26] graph: add unit test case
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (11 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 12/26] graph: implement fastpath API routines jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 14/26] graph: add performance testcase jerinj
` (13 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark,
pbhagavatula, ndabilpuram
From: Kiran Kumar K <kirankumark@marvell.com>
Adding the unit test to test the functionality of node and graph APIs.
Testing includes registering a node, cloning a node, creating a graph,
perform graph walk, collecting stats and all node and graph debug APIs.
example command to test:
echo "graph_autotest" | sudo ./build/app/test/dpdk-test -c 0x30
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
app/test/Makefile | 4 +
app/test/meson.build | 10 +-
app/test/test_graph.c | 820 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 832 insertions(+), 2 deletions(-)
create mode 100644 app/test/test_graph.c
diff --git a/app/test/Makefile b/app/test/Makefile
index 1f080d162..065582916 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -221,6 +221,10 @@ SRCS-y += test_event_timer_adapter.c
SRCS-y += test_event_crypto_adapter.c
endif
+ifeq ($(CONFIG_RTE_LIBRTE_GRAPH), y)
+SRCS-y += test_graph.c
+endif
+
ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
SRCS-y += test_rawdev.c
endif
diff --git a/app/test/meson.build b/app/test/meson.build
index 0a2ce710f..855403932 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -51,6 +51,7 @@ test_sources = files('commands.c',
'test_fib6_perf.c',
'test_func_reentrancy.c',
'test_flow_classify.c',
+ 'test_graph.c',
'test_hash.c',
'test_hash_functions.c',
'test_hash_multiwriter.c',
@@ -151,7 +152,9 @@ test_deps = ['acl',
'rib',
'ring',
'stack',
- 'timer'
+ 'timer',
+ 'graph',
+ 'node'
]
fast_test_names = [
@@ -183,6 +186,7 @@ fast_test_names = [
'fib6_autotest',
'func_reentrancy_autotest',
'flow_classify_autotest',
+ 'graph_autotest',
'hash_autotest',
'interrupt_autotest',
'logs_autotest',
@@ -383,13 +387,15 @@ endforeach
test_dep_objs += cc.find_library('execinfo', required: false)
link_libs = []
+link_nodes = []
if get_option('default_library') == 'static'
link_libs = dpdk_drivers
+ link_nodes = dpdk_graph_nodes
endif
dpdk_test = executable('dpdk-test',
test_sources,
- link_whole: link_libs,
+ link_whole: link_libs + link_nodes,
dependencies: test_dep_objs,
c_args: [cflags, '-DALLOW_EXPERIMENTAL_API'],
install_rpath: driver_install_path,
diff --git a/app/test/test_graph.c b/app/test/test_graph.c
new file mode 100644
index 000000000..e0116a9ce
--- /dev/null
+++ b/app/test/test_graph.c
@@ -0,0 +1,820 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#include <assert.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_errno.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_mbuf.h>
+
+#include "test.h"
+
+static uint16_t test_node_worker_source(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+static uint16_t test_node0_worker(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+static uint16_t test_node1_worker(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+static uint16_t test_node2_worker(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+static uint16_t test_node3_worker(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+#define MBUFF_SIZE 512
+#define MAX_NODES 4
+
+static struct rte_mbuf mbuf[MAX_NODES + 1][MBUFF_SIZE];
+static void *mbuf_p[MAX_NODES + 1][MBUFF_SIZE];
+static rte_graph_t graph_id;
+static uint64_t obj_stats[MAX_NODES + 1];
+static uint64_t fn_calls[MAX_NODES + 1];
+
+const char *node_patterns[] = {
+ "test_node_source1", "test_node00",
+ "test_node00-test_node11", "test_node00-test_node22",
+ "test_node00-test_node33",
+};
+
+const char *node_names[] = {
+ "test_node00",
+ "test_node00-test_node11",
+ "test_node00-test_node22",
+ "test_node00-test_node33",
+};
+
+struct test_node_register {
+#define NODE_NAMESIZE 64
+ char name[NODE_NAMESIZE];
+ rte_node_process_t process;
+ uint16_t nb_edges;
+ const char *next_nodes[MAX_NODES];
+};
+
+typedef struct {
+ uint32_t idx;
+ struct test_node_register node;
+} test_node_t;
+
+typedef struct {
+ test_node_t test_node[MAX_NODES];
+} test_main_t;
+
+static test_main_t test_main = {
+ .test_node = {
+ {
+ .node = {
+ .name = "test_node00",
+ .process = test_node0_worker,
+ .nb_edges = 2,
+ .next_nodes = {"test_node00-"
+ "test_node11",
+ "test_node00-"
+ "test_node22"},
+ },
+ },
+ {
+ .node = {
+ .name = "test_node11",
+ .process = test_node1_worker,
+ .nb_edges = 1,
+ .next_nodes = {"test_node00-"
+ "test_node22"},
+ },
+ },
+ {
+ .node = {
+ .name = "test_node22",
+ .process = test_node2_worker,
+ .nb_edges = 1,
+ .next_nodes = {"test_node00-"
+ "test_node33"},
+ },
+ },
+ {
+ .node = {
+ .name = "test_node33",
+ .process = test_node3_worker,
+ .nb_edges = 1,
+ .next_nodes = {"test_node00"},
+ },
+ },
+ },
+};
+
+static int
+node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ RTE_SET_USED(graph);
+ *(uint32_t *)node->ctx = node->id;
+
+ return 0;
+}
+
+static struct rte_node_register test_node_source = {
+ .name = "test_node_source1",
+ .process = test_node_worker_source,
+ .flags = RTE_NODE_SOURCE_F,
+ .nb_edges = 2,
+ .init = node_init,
+ .next_nodes = {"test_node00", "test_node00-test_node11"},
+};
+RTE_NODE_REGISTER(test_node_source);
+
+static struct rte_node_register test_node0 = {
+ .name = "test_node00",
+ .process = test_node0_worker,
+ .init = node_init,
+};
+RTE_NODE_REGISTER(test_node0);
+
+uint16_t
+test_node_worker_source(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ uint32_t obj_node0 = rand() % 100, obj_node1;
+ test_main_t *tm = &test_main;
+ struct rte_mbuf *data;
+ void **next_stream;
+ rte_node_t next;
+ uint32_t i;
+
+ RTE_SET_USED(objs);
+ nb_objs = RTE_GRAPH_BURST_SIZE;
+
+ /* Prepare stream for next node 0 */
+ obj_node0 = nb_objs * obj_node0 * 0.01;
+ next = 0;
+ next_stream = rte_node_next_stream_get(graph, node, next, obj_node0);
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[0][i];
+ data->udata64 = ((uint64_t)tm->test_node[0].idx << 32) | i;
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ next_stream[i] = &mbuf[0][i];
+ }
+ rte_node_next_stream_put(graph, node, next, obj_node0);
+
+ /* Prepare stream for next node 1 */
+ obj_node1 = nb_objs - obj_node0;
+ next = 1;
+ next_stream = rte_node_next_stream_get(graph, node, next, obj_node1);
+ for (i = 0; i < obj_node1; i++) {
+ data = &mbuf[0][obj_node0 + i];
+ data->udata64 = ((uint64_t)tm->test_node[1].idx << 32) | i;
+ if ((i + 1) == obj_node1)
+ data->udata64 |= (1 << 16);
+ next_stream[i] = &mbuf[0][obj_node0 + i];
+ }
+
+ rte_node_next_stream_put(graph, node, next, obj_node1);
+ obj_stats[0] += nb_objs;
+ fn_calls[0] += 1;
+ return nb_objs;
+}
+
+uint16_t
+test_node0_worker(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ test_main_t *tm = &test_main;
+
+ if (*(uint32_t *)node->ctx == test_node0.id) {
+ uint32_t obj_node0 = rand() % 100, obj_node1;
+ struct rte_mbuf *data;
+ uint8_t second_pass = 0;
+ uint32_t count = 0;
+ uint32_t i;
+
+ obj_stats[1] += nb_objs;
+ fn_calls[1] += 1;
+
+ for (i = 0; i < nb_objs; i++) {
+ data = (struct rte_mbuf *)objs[i];
+ if ((data->udata64 >> 32) != tm->test_node[0].idx) {
+ printf("Data idx miss match at node 0, expected"
+ " = %u got = %u\n",
+ tm->test_node[0].idx,
+ (uint32_t)(data->udata64 >> 32));
+ goto end;
+ }
+
+ if ((data->udata64 & 0xffff) != (i - count)) {
+ printf("Expected buff count miss match at "
+ "node 0\n");
+ goto end;
+ }
+
+ if (data->udata64 & (0x1 << 16))
+ count = i + 1;
+ if (data->udata64 & (0x1 << 17))
+ second_pass = 1;
+ }
+
+ if (count != i) {
+ printf("Count mismatch at node 0\n");
+ goto end;
+ }
+
+ obj_node0 = nb_objs * obj_node0 * 0.01;
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[1][i];
+ data->udata64 =
+ ((uint64_t)tm->test_node[1].idx << 32) | i;
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ if (second_pass)
+ data->udata64 |= (1 << 17);
+ }
+ rte_node_enqueue(graph, node, 0, (void **)&mbuf_p[1][0],
+ obj_node0);
+
+ obj_node1 = nb_objs - obj_node0;
+ for (i = 0; i < obj_node1; i++) {
+ data = &mbuf[1][obj_node0 + i];
+ data->udata64 =
+ ((uint64_t)tm->test_node[2].idx << 32) | i;
+ if ((i + 1) == obj_node1)
+ data->udata64 |= (1 << 16);
+ if (second_pass)
+ data->udata64 |= (1 << 17);
+ }
+ rte_node_enqueue(graph, node, 1, (void **)&mbuf_p[1][obj_node0],
+ obj_node1);
+
+ } else if (*(uint32_t *)node->ctx == tm->test_node[1].idx) {
+ test_node1_worker(graph, node, objs, nb_objs);
+ } else if (*(uint32_t *)node->ctx == tm->test_node[2].idx) {
+ test_node2_worker(graph, node, objs, nb_objs);
+ } else if (*(uint32_t *)node->ctx == tm->test_node[3].idx) {
+ test_node3_worker(graph, node, objs, nb_objs);
+ } else {
+ printf("Unexpected node context\n");
+ }
+
+end:
+ return nb_objs;
+}
+
+uint16_t
+test_node1_worker(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ test_main_t *tm = &test_main;
+ uint8_t second_pass = 0;
+ uint32_t obj_node0 = 0;
+ struct rte_mbuf *data;
+ uint32_t count = 0;
+ uint32_t i;
+
+ obj_stats[2] += nb_objs;
+ fn_calls[2] += 1;
+ for (i = 0; i < nb_objs; i++) {
+ data = (struct rte_mbuf *)objs[i];
+ if ((data->udata64 >> 32) != tm->test_node[1].idx) {
+ printf("Data idx miss match at node 1, expected = %u"
+ " got = %u\n",
+ tm->test_node[1].idx,
+ (uint32_t)(data->udata64 >> 32));
+ goto end;
+ }
+
+ if ((data->udata64 & 0xffff) != (i - count)) {
+ printf("Expected buff count miss match at node 1\n");
+ goto end;
+ }
+
+ if (data->udata64 & (0x1 << 16))
+ count = i + 1;
+ if (data->udata64 & (0x1 << 17))
+ second_pass = 1;
+ }
+
+ if (count != i) {
+ printf("Count mismatch at node 1\n");
+ goto end;
+ }
+
+ obj_node0 = nb_objs;
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[2][i];
+ data->udata64 = ((uint64_t)tm->test_node[2].idx << 32) | i;
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ if (second_pass)
+ data->udata64 |= (1 << 17);
+ }
+ rte_node_enqueue(graph, node, 0, (void **)&mbuf_p[2][0], obj_node0);
+
+end:
+ return nb_objs;
+}
+
+uint16_t
+test_node2_worker(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ test_main_t *tm = &test_main;
+ uint8_t second_pass = 0;
+ struct rte_mbuf *data;
+ uint32_t count = 0;
+ uint32_t obj_node0;
+ uint32_t i;
+
+ obj_stats[3] += nb_objs;
+ fn_calls[3] += 1;
+ for (i = 0; i < nb_objs; i++) {
+ data = (struct rte_mbuf *)objs[i];
+ if ((data->udata64 >> 32) != tm->test_node[2].idx) {
+ printf("Data idx miss match at node 2, expected = %u"
+ " got = %u\n",
+ tm->test_node[2].idx,
+ (uint32_t)(data->udata64 >> 32));
+ goto end;
+ }
+
+ if ((data->udata64 & 0xffff) != (i - count)) {
+ printf("Expected buff count miss match at node 2\n");
+ goto end;
+ }
+
+ if (data->udata64 & (0x1 << 16))
+ count = i + 1;
+ if (data->udata64 & (0x1 << 17))
+ second_pass = 1;
+ }
+
+ if (count != i) {
+ printf("Count mismatch at node 2\n");
+ goto end;
+ }
+
+ if (!second_pass) {
+ obj_node0 = nb_objs;
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[3][i];
+ data->udata64 =
+ ((uint64_t)tm->test_node[3].idx << 32) | i;
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ }
+ rte_node_enqueue(graph, node, 0, (void **)&mbuf_p[3][0],
+ obj_node0);
+ }
+
+end:
+ return nb_objs;
+}
+
+uint16_t
+test_node3_worker(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ test_main_t *tm = &test_main;
+ uint8_t second_pass = 0;
+ struct rte_mbuf *data;
+ uint32_t count = 0;
+ uint32_t obj_node0;
+ uint32_t i;
+
+ obj_stats[4] += nb_objs;
+ fn_calls[4] += 1;
+ for (i = 0; i < nb_objs; i++) {
+ data = (struct rte_mbuf *)objs[i];
+ if ((data->udata64 >> 32) != tm->test_node[3].idx) {
+ printf("Data idx miss match at node 3, expected = %u"
+ " got = %u\n",
+ tm->test_node[3].idx,
+ (uint32_t)(data->udata64 >> 32));
+ goto end;
+ }
+
+ if ((data->udata64 & 0xffff) != (i - count)) {
+ printf("Expected buff count miss match at node 3\n");
+ goto end;
+ }
+
+ if (data->udata64 & (0x1 << 16))
+ count = i + 1;
+ if (data->udata64 & (0x1 << 17))
+ second_pass = 1;
+ }
+
+ if (count != i) {
+ printf("Count mismatch at node 3\n");
+ goto end;
+ }
+
+ if (second_pass) {
+ printf("Unexpected buffers are at node 3\n");
+ goto end;
+ } else {
+ obj_node0 = nb_objs * 2;
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[4][i];
+ data->udata64 =
+ ((uint64_t)tm->test_node[0].idx << 32) | i;
+ data->udata64 |= (1 << 17);
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ }
+ rte_node_enqueue(graph, node, 0, (void **)&mbuf_p[4][0],
+ obj_node0);
+ }
+
+end:
+ return nb_objs;
+}
+
+static int
+test_lookup_functions(void)
+{
+ test_main_t *tm = &test_main;
+ int i;
+
+ /* Verify the name with ID */
+ for (i = 1; i < MAX_NODES; i++) {
+ char *name = rte_node_id_to_name(tm->test_node[i].idx);
+ if (strcmp(name, node_names[i]) != 0) {
+ printf("Test node name verify by ID = %d failed "
+ "Expected = %s, got %s\n",
+ i, node_names[i], name);
+ return -1;
+ }
+ }
+
+ /* Verify by name */
+ for (i = 1; i < MAX_NODES; i++) {
+ uint32_t idx = rte_node_from_name(node_names[i]);
+ if (idx != tm->test_node[i].idx) {
+ printf("Test node ID verify by name = %s failed "
+ "Expected = %d, got %d\n",
+ node_names[i], tm->test_node[i].idx, idx);
+ return -1;
+ }
+ }
+
+ /* Verify edge count */
+ for (i = 1; i < MAX_NODES; i++) {
+ uint32_t count = rte_node_edge_count(tm->test_node[i].idx);
+ if (count != tm->test_node[i].node.nb_edges) {
+ printf("Test number of edges for node = %s failed Expected = %d, got = %d\n",
+ tm->test_node[i].node.name,
+ tm->test_node[i].node.nb_edges, count);
+ return -1;
+ }
+ }
+
+ /* Verify edge names */
+ for (i = 1; i < MAX_NODES; i++) {
+ uint32_t j, count;
+ char **next_edges;
+
+ count = rte_node_edge_get(tm->test_node[i].idx, NULL);
+ if (count != tm->test_node[i].node.nb_edges * sizeof(char *)) {
+ printf("Test number of edge count for node = %s failed Expected = %d, got = %d\n",
+ tm->test_node[i].node.name,
+ tm->test_node[i].node.nb_edges, count);
+ return -1;
+ }
+ next_edges = malloc(count);
+ count = rte_node_edge_get(tm->test_node[i].idx, next_edges);
+ if (count != tm->test_node[i].node.nb_edges) {
+ printf("Test number of edges for node = %s failed Expected = %d, got %d\n",
+ tm->test_node[i].node.name,
+ tm->test_node[i].node.nb_edges, count);
+ return -1;
+ }
+
+ for (j = 0; j < count; j++) {
+ if (strcmp(next_edges[j],
+ tm->test_node[i].node.next_nodes[j]) != 0) {
+ printf("Edge name miss match, expected = %s got = %s\n",
+ tm->test_node[i].node.next_nodes[j],
+ next_edges[j]);
+ return -1;
+ }
+ }
+ free(next_edges);
+ }
+
+ return 0;
+}
+
+static int
+test_node_clone(void)
+{
+ test_main_t *tm = &test_main;
+ uint32_t node_id, dummy_id;
+ int i;
+
+ node_id = rte_node_from_name("test_node00");
+ tm->test_node[0].idx = node_id;
+
+ /* Clone with same name, should fail */
+ dummy_id = rte_node_clone(node_id, "test_node00");
+ if (!rte_node_is_invalid(dummy_id)) {
+ printf("Got valid id when clone with same name, Expecting fail\n");
+ return -1;
+ }
+
+ for (i = 1; i < MAX_NODES; i++) {
+ tm->test_node[i].idx =
+ rte_node_clone(node_id, tm->test_node[i].node.name);
+ if (rte_node_is_invalid(tm->test_node[i].idx)) {
+ printf("Got invalid node id\n");
+ return -1;
+ }
+ }
+
+ /* Clone from cloned node should fail */
+ dummy_id = rte_node_clone(tm->test_node[1].idx, "dummy_node");
+ if (!rte_node_is_invalid(dummy_id)) {
+ printf("Got valid node id when cloning from cloned node, expected fail\n");
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+test_update_edges(void)
+{
+ test_main_t *tm = &test_main;
+ uint32_t node_id;
+ uint16_t count;
+ int i;
+
+ node_id = rte_node_from_name("test_node00");
+ count = rte_node_edge_update(node_id, 0,
+ tm->test_node[0].node.next_nodes,
+ tm->test_node[0].node.nb_edges);
+ if (count != tm->test_node[0].node.nb_edges) {
+ printf("Update edges failed expected: %d got = %d\n",
+ tm->test_node[0].node.nb_edges, count);
+ return -1;
+ }
+
+ for (i = 1; i < MAX_NODES; i++) {
+ count = rte_node_edge_update(tm->test_node[i].idx, 0,
+ tm->test_node[i].node.next_nodes,
+ tm->test_node[i].node.nb_edges);
+ if (count != tm->test_node[i].node.nb_edges) {
+ printf("Update edges failed expected: %d got = %d\n",
+ tm->test_node[i].node.nb_edges, count);
+ return -1;
+ }
+
+ count = rte_node_edge_shrink(tm->test_node[i].idx,
+ tm->test_node[i].node.nb_edges);
+ if (count != tm->test_node[i].node.nb_edges) {
+ printf("Shrink edges failed\n");
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+test_create_graph(void)
+{
+ static const char *node_patterns_dummy[] = {
+ "test_node_source1", "test_node00",
+ "test_node00-test_node11", "test_node00-test_node22",
+ "test_node00-test_node33", "test_node00-dummy_node",
+ };
+ struct rte_graph_param gconf = {
+ .socket_id = SOCKET_ID_ANY,
+ .nb_node_patterns = 6,
+ .node_patterns = node_patterns_dummy,
+ };
+ uint32_t dummy_node_id;
+ uint32_t node_id;
+
+ node_id = rte_node_from_name("test_node00");
+ dummy_node_id = rte_node_clone(node_id, "dummy_node");
+ if (rte_node_is_invalid(dummy_node_id)) {
+ printf("Got invalid node id\n");
+ return -1;
+ }
+
+ graph_id = rte_graph_create("worker0", &gconf);
+ if (graph_id != RTE_GRAPH_ID_INVALID) {
+ printf("Graph creation success with isolated node, expected graph creation fail\n");
+ return -1;
+ }
+
+ gconf.nb_node_patterns = 5;
+ gconf.node_patterns = node_patterns;
+ graph_id = rte_graph_create("worker0", &gconf);
+ if (graph_id == RTE_GRAPH_ID_INVALID) {
+ printf("Graph creation failed with error = %d\n", rte_errno);
+ return -1;
+ }
+ return 0;
+}
+
+static int
+test_graph_walk(void)
+{
+ struct rte_graph *graph = rte_graph_lookup("worker0");
+ int i;
+
+ if (!graph) {
+ printf("Graph lookup failed\n");
+ return -1;
+ }
+
+ for (i = 0; i < 5; i++)
+ rte_graph_walk(graph);
+ return 0;
+}
+
+static int
+test_graph_lookup_functions(void)
+{
+ test_main_t *tm = &test_main;
+ struct rte_node *node;
+ int i;
+
+ for (i = 0; i < MAX_NODES; i++) {
+ node = rte_graph_node_get(graph_id, tm->test_node[i].idx);
+ if (!node) {
+ printf("rte_graph_node_get, failed for node = %d\n",
+ tm->test_node[i].idx);
+ return -1;
+ }
+
+ if (tm->test_node[i].idx != node->id) {
+ printf("Node id didn't match, expected = %d got = %d\n",
+ tm->test_node[i].idx, node->id);
+ return 0;
+ }
+
+ if (strncmp(node->name, node_names[i], RTE_NODE_NAMESIZE)) {
+ printf("Node name didn't match, expected = %s got %s\n",
+ node_names[i], node->name);
+ return -1;
+ }
+ }
+
+ for (i = 0; i < MAX_NODES; i++) {
+ node = rte_graph_node_get_by_name("worker0", node_names[i]);
+ if (!node) {
+ printf("rte_graph_node_get, failed for node = %d\n",
+ tm->test_node[i].idx);
+ return -1;
+ }
+
+ if (tm->test_node[i].idx != node->id) {
+ printf("Node id didn't match, expected = %d got = %d\n",
+ tm->test_node[i].idx, node->id);
+ return 0;
+ }
+
+ if (strncmp(node->name, node_names[i], RTE_NODE_NAMESIZE)) {
+ printf("Node name didn't match, expected = %s got %s\n",
+ node_names[i], node->name);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+graph_cluster_stats_cb_t(bool is_first, bool is_last, void *cookie,
+ const struct rte_graph_cluster_node_stats *st)
+{
+ int i;
+
+ RTE_SET_USED(is_first);
+ RTE_SET_USED(is_last);
+ RTE_SET_USED(cookie);
+
+ for (i = 0; i < MAX_NODES + 1; i++) {
+ rte_node_t id = rte_node_from_name(node_patterns[i]);
+ if (id == st->id) {
+ if (obj_stats[i] != st->objs) {
+ printf("Obj count miss match for node = %s expected = %"PRId64", got=%"PRId64"\n",
+ node_patterns[i], obj_stats[i],
+ st->objs);
+ return -1;
+ }
+
+ if (fn_calls[i] != st->calls) {
+ printf("Func call miss match for node = %s expected = %"PRId64", got = %"PRId64"\n",
+ node_patterns[i], fn_calls[i],
+ st->calls);
+ return -1;
+ }
+ }
+ }
+ return 0;
+}
+
+static int
+test_print_stats(void)
+{
+ struct rte_graph_cluster_stats_param s_param;
+ struct rte_graph_cluster_stats *stats;
+ const char *pattern = "worker0";
+
+ if (!rte_graph_has_stats_feature())
+ return 0;
+
+ /* Prepare stats object */
+ memset(&s_param, 0, sizeof(s_param));
+ s_param.f = stdout;
+ s_param.socket_id = SOCKET_ID_ANY;
+ s_param.graph_patterns = &pattern;
+ s_param.nb_graph_patterns = 1;
+ s_param.fn = graph_cluster_stats_cb_t;
+
+ stats = rte_graph_cluster_stats_create(&s_param);
+ if (stats == NULL) {
+ printf("Unable to get stats\n");
+ return -1;
+ }
+ /* Clear screen and move to top left */
+ rte_graph_cluster_stats_get(stats, 0);
+ rte_graph_cluster_stats_destroy(stats);
+
+ return 0;
+}
+
+static int
+graph_setup(void)
+{
+ int i, j;
+
+ for (i = 0; i <= MAX_NODES; i++) {
+ for (j = 0; j < MBUFF_SIZE; j++)
+ mbuf_p[i][j] = &mbuf[i][j];
+ }
+ if (test_node_clone()) {
+ printf("test_node_clone: fail\n");
+ return -1;
+ }
+ printf("test_node_clone: pass\n");
+
+ return 0;
+}
+
+static void
+graph_teardown(void)
+{
+ rte_graph_t id;
+
+ id = rte_graph_destroy("worker0");
+ if (id == RTE_GRAPH_ID_INVALID)
+ printf("Graph Destroy failed\n");
+}
+
+static struct unit_test_suite graph_testsuite = {
+ .suite_name = "Graph library test suite",
+ .setup = graph_setup,
+ .teardown = graph_teardown,
+ .unit_test_cases = {
+ TEST_CASE(test_update_edges),
+ TEST_CASE(test_lookup_functions),
+ TEST_CASE(test_create_graph),
+ TEST_CASE(test_graph_lookup_functions),
+ TEST_CASE(test_graph_walk),
+ TEST_CASE(test_print_stats),
+ TEST_CASES_END(), /**< NULL terminate unit test array */
+ },
+};
+
+static int
+graph_autotest_fn(void)
+{
+ return unit_test_suite_runner(&graph_testsuite);
+}
+
+REGISTER_TEST_COMMAND(graph_autotest, graph_autotest_fn);
+
+static int
+test_node_list_dump(void)
+{
+ rte_node_list_dump(stdout);
+
+ return TEST_SUCCESS;
+}
+REGISTER_TEST_COMMAND(node_list_dump, test_node_list_dump);
+
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 14/26] graph: add performance testcase
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (12 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 13/26] graph: add unit test case jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 15/26] node: add log infra and null node jerinj
` (12 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark,
pbhagavatula, ndabilpuram
From: Pavan Nikhilesh <pbhagavatula@marvell.com>
Add unit test framework to create and test performance of various graph
models.
example command to test:
echo "graph_perf_autotest" | sudo ./build/app/test/dpdk-test -c 0x30
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
app/test/Makefile | 1 +
app/test/meson.build | 1 +
app/test/test_graph_perf.c | 1057 ++++++++++++++++++++++++++++++++++++
3 files changed, 1059 insertions(+)
create mode 100644 app/test/test_graph_perf.c
diff --git a/app/test/Makefile b/app/test/Makefile
index 065582916..cd2801427 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -223,6 +223,7 @@ endif
ifeq ($(CONFIG_RTE_LIBRTE_GRAPH), y)
SRCS-y += test_graph.c
+SRCS-y += test_graph_perf.c
endif
ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
diff --git a/app/test/meson.build b/app/test/meson.build
index 855403932..3fe155778 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -52,6 +52,7 @@ test_sources = files('commands.c',
'test_func_reentrancy.c',
'test_flow_classify.c',
'test_graph.c',
+ 'test_graph_perf.c',
'test_hash.c',
'test_hash_functions.c',
'test_hash_multiwriter.c',
diff --git a/app/test/test_graph_perf.c b/app/test/test_graph_perf.c
new file mode 100644
index 000000000..a629f1e35
--- /dev/null
+++ b/app/test/test_graph_perf.c
@@ -0,0 +1,1057 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#include <inttypes.h>
+#include <signal.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_errno.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_lcore.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+
+#include "test.h"
+
+#define TEST_GRAPH_PERF_MZ "graph_perf_data"
+#define TEST_GRAPH_SRC_NAME "test_graph_perf_source"
+#define TEST_GRAPH_SRC_BRST_ONE_NAME "test_graph_perf_source_one"
+#define TEST_GRAPH_WRK_NAME "test_graph_perf_worker"
+#define TEST_GRAPH_SNK_NAME "test_graph_perf_sink"
+
+#define SOURCES(map) RTE_DIM(map)
+#define STAGES(map) RTE_DIM(map)
+#define NODES_PER_STAGE(map) RTE_DIM(map[0])
+#define SINKS(map) RTE_DIM(map[0])
+
+#define MAX_EDGES_PER_NODE 7
+
+struct test_node_data {
+ uint8_t node_id;
+ uint8_t is_sink;
+ uint8_t next_nodes[MAX_EDGES_PER_NODE];
+ uint8_t next_percentage[MAX_EDGES_PER_NODE];
+};
+
+struct test_graph_perf {
+ uint16_t nb_nodes;
+ rte_graph_t graph_id;
+ struct test_node_data *node_data;
+};
+
+struct graph_lcore_data {
+ uint8_t done;
+ rte_graph_t graph_id;
+};
+
+static struct test_node_data *
+graph_get_node_data(struct test_graph_perf *graph_data, rte_node_t id)
+{
+ struct test_node_data *node_data = NULL;
+ int i;
+
+ for (i = 0; i < graph_data->nb_nodes; i++)
+ if (graph_data->node_data[i].node_id == id) {
+ node_data = &graph_data->node_data[i];
+ break;
+ }
+
+ return node_data;
+}
+
+static int
+test_node_ctx_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ struct test_graph_perf *graph_data;
+ struct test_node_data *node_data;
+ const struct rte_memzone *mz;
+ rte_node_t nid = node->id;
+ rte_edge_t edge = 0;
+ int i;
+
+ RTE_SET_USED(graph);
+
+ mz = rte_memzone_lookup(TEST_GRAPH_PERF_MZ);
+ graph_data = mz->addr;
+ node_data = graph_get_node_data(graph_data, nid);
+ node->ctx[0] = node->nb_edges;
+ for (i = 0; i < node->nb_edges && !node_data->is_sink; i++, edge++) {
+ node->ctx[i + 1] = edge;
+ node->ctx[i + 9] = node_data->next_percentage[i];
+ }
+
+ return 0;
+}
+
+/* Source node function */
+static uint16_t
+test_perf_node_worker_source(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ uint16_t count;
+ int i;
+
+ RTE_SET_USED(objs);
+ RTE_SET_USED(nb_objs);
+
+ /* Create a proportional stream for every next */
+ for (i = 0; i < node->ctx[0]; i++) {
+ count = (node->ctx[i + 9] * RTE_GRAPH_BURST_SIZE) / 100;
+ rte_node_next_stream_get(graph, node, node->ctx[i + 1], count);
+ rte_node_next_stream_put(graph, node, node->ctx[i + 1], count);
+ }
+
+ return RTE_GRAPH_BURST_SIZE;
+}
+
+static struct rte_node_register test_graph_perf_source = {
+ .name = TEST_GRAPH_SRC_NAME,
+ .process = test_perf_node_worker_source,
+ .flags = RTE_NODE_SOURCE_F,
+ .init = test_node_ctx_init,
+};
+
+RTE_NODE_REGISTER(test_graph_perf_source);
+
+static uint16_t
+test_perf_node_worker_source_burst_one(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ uint16_t count;
+ int i;
+
+ RTE_SET_USED(objs);
+ RTE_SET_USED(nb_objs);
+
+ /* Create a proportional stream for every next */
+ for (i = 0; i < node->ctx[0]; i++) {
+ count = (node->ctx[i + 9]) / 100;
+ rte_node_next_stream_get(graph, node, node->ctx[i + 1], count);
+ rte_node_next_stream_put(graph, node, node->ctx[i + 1], count);
+ }
+
+ return 1;
+}
+
+static struct rte_node_register test_graph_perf_source_burst_one = {
+ .name = TEST_GRAPH_SRC_BRST_ONE_NAME,
+ .process = test_perf_node_worker_source_burst_one,
+ .flags = RTE_NODE_SOURCE_F,
+ .init = test_node_ctx_init,
+};
+
+RTE_NODE_REGISTER(test_graph_perf_source_burst_one);
+
+/* Worker node function */
+static uint16_t
+test_perf_node_worker(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ uint16_t next = 0;
+ uint16_t enq = 0;
+ uint16_t count;
+ int i;
+
+ /* Move stream for single next node */
+ if (node->ctx[0] == 1) {
+ rte_node_next_stream_move(graph, node, node->ctx[1]);
+ return nb_objs;
+ }
+
+ /* Enqueue objects to next nodes proportionally */
+ for (i = 0; i < node->ctx[0]; i++) {
+ next = node->ctx[i + 1];
+ count = (node->ctx[i + 9] * nb_objs) / 100;
+ enq += count;
+ while (count) {
+ switch (count & (4 - 1)) {
+ case 0:
+ rte_node_enqueue_x4(graph, node, next, objs[0],
+ objs[1], objs[2], objs[3]);
+ objs += 4;
+ count -= 4;
+ break;
+ case 1:
+ rte_node_enqueue_x1(graph, node, next, objs[0]);
+ objs += 1;
+ count -= 1;
+ break;
+ case 2:
+ rte_node_enqueue_x2(graph, node, next, objs[0],
+ objs[1]);
+ objs += 2;
+ count -= 2;
+ break;
+ case 3:
+ rte_node_enqueue_x2(graph, node, next, objs[0],
+ objs[1]);
+ rte_node_enqueue_x1(graph, node, next, objs[0]);
+ objs += 3;
+ count -= 3;
+ break;
+ }
+ }
+ }
+
+ if (enq != nb_objs)
+ rte_node_enqueue(graph, node, next, objs, nb_objs - enq);
+
+ return nb_objs;
+}
+
+static struct rte_node_register test_graph_perf_worker = {
+ .name = TEST_GRAPH_WRK_NAME,
+ .process = test_perf_node_worker,
+ .init = test_node_ctx_init,
+};
+
+RTE_NODE_REGISTER(test_graph_perf_worker);
+
+/* Last node in graph a.k.a sink node */
+static uint16_t
+test_perf_node_sink(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ RTE_SET_USED(graph);
+ RTE_SET_USED(node);
+ RTE_SET_USED(objs);
+ RTE_SET_USED(nb_objs);
+
+ return nb_objs;
+}
+
+static struct rte_node_register test_graph_perf_sink = {
+ .name = TEST_GRAPH_SNK_NAME,
+ .process = test_perf_node_sink,
+ .init = test_node_ctx_init,
+};
+
+RTE_NODE_REGISTER(test_graph_perf_sink);
+
+static int
+graph_perf_setup(void)
+{
+ if (rte_lcore_count() < 2) {
+ printf("Test requires at least 2 lcores\n");
+ return TEST_SKIPPED;
+ }
+
+ return 0;
+}
+
+static void
+graph_perf_teardown(void)
+{
+}
+
+static inline rte_node_t
+graph_node_get(const char *pname, char *nname)
+{
+ rte_node_t pnode_id = rte_node_from_name(pname);
+ char lookup_name[RTE_NODE_NAMESIZE];
+ rte_node_t node_id;
+
+ snprintf(lookup_name, RTE_NODE_NAMESIZE, "%s-%s", pname, nname);
+ node_id = rte_node_from_name(lookup_name);
+
+ if (node_id != RTE_NODE_ID_INVALID) {
+ if (rte_node_edge_count(node_id))
+ rte_node_edge_shrink(node_id, 0);
+ return node_id;
+ }
+
+ return rte_node_clone(pnode_id, nname);
+}
+
+static uint16_t
+graph_node_count_edges(uint32_t stage, uint16_t node, uint16_t nodes_per_stage,
+ uint8_t edge_map[][nodes_per_stage][nodes_per_stage],
+ char *ename[], struct test_node_data *node_data,
+ rte_node_t **node_map)
+{
+ uint8_t total_percent = 0;
+ uint16_t edges = 0;
+ int i;
+
+ for (i = 0; i < nodes_per_stage && edges < MAX_EDGES_PER_NODE; i++) {
+ if (edge_map[stage + 1][i][node]) {
+ ename[edges] = malloc(sizeof(char) * RTE_NODE_NAMESIZE);
+ snprintf(ename[edges], RTE_NODE_NAMESIZE, "%s",
+ rte_node_id_to_name(node_map[stage + 1][i]));
+ node_data->next_nodes[edges] = node_map[stage + 1][i];
+ node_data->next_percentage[edges] =
+ edge_map[stage + 1][i][node];
+ edges++;
+ total_percent += edge_map[stage + 1][i][node];
+ }
+ }
+
+ if (edges >= MAX_EDGES_PER_NODE || (edges && total_percent != 100)) {
+ for (i = 0; i < edges; i++)
+ free(ename[i]);
+ return RTE_EDGE_ID_INVALID;
+ }
+
+ return edges;
+}
+
+static int
+graph_init(const char *gname, uint8_t nb_srcs, uint8_t nb_sinks,
+ uint32_t stages, uint16_t nodes_per_stage,
+ uint8_t src_map[][nodes_per_stage], uint8_t snk_map[][nb_sinks],
+ uint8_t edge_map[][nodes_per_stage][nodes_per_stage],
+ uint8_t burst_one)
+{
+ struct test_graph_perf *graph_data;
+ char nname[RTE_NODE_NAMESIZE / 2];
+ struct test_node_data *node_data;
+ char *ename[nodes_per_stage];
+ struct rte_graph_param gconf;
+ const struct rte_memzone *mz;
+ uint8_t total_percent = 0;
+ rte_node_t *src_nodes;
+ rte_node_t *snk_nodes;
+ rte_node_t **node_map;
+ char **node_patterns;
+ rte_graph_t graph_id;
+ rte_edge_t edges;
+ rte_edge_t count;
+ uint32_t i, j, k;
+
+ mz = rte_memzone_reserve(TEST_GRAPH_PERF_MZ,
+ sizeof(struct test_graph_perf), 0, 0);
+ if (mz == NULL) {
+ printf("Failed to allocate graph common memory\n");
+ return -ENOMEM;
+ }
+
+ graph_data = mz->addr;
+ graph_data->nb_nodes = 0;
+ graph_data->node_data =
+ malloc(sizeof(struct test_node_data) *
+ (nb_srcs + nb_sinks + stages * nodes_per_stage));
+ if (graph_data->node_data == NULL) {
+ printf("Failed to reserve memzone for graph data\n");
+ goto memzone_free;
+ }
+
+ node_patterns = malloc(sizeof(char *) *
+ (nb_srcs + nb_sinks + stages * nodes_per_stage));
+ if (node_patterns == NULL) {
+ printf("Failed to reserve memory for node patterns\n");
+ goto data_free;
+ }
+
+ src_nodes = malloc(sizeof(rte_node_t) * nb_srcs);
+ if (src_nodes == NULL) {
+ printf("Failed to reserve memory for src nodes\n");
+ goto pattern_free;
+ }
+
+ snk_nodes = malloc(sizeof(rte_node_t) * nb_sinks);
+ if (snk_nodes == NULL) {
+ printf("Failed to reserve memory for snk nodes\n");
+ goto src_free;
+ }
+
+ node_map = malloc(sizeof(rte_node_t *) * stages +
+ sizeof(rte_node_t) * nodes_per_stage * stages);
+ if (node_map == NULL) {
+ printf("Failed to reserve memory for node map\n");
+ goto snk_free;
+ }
+
+ /* Setup the Graph */
+ for (i = 0; i < stages; i++) {
+ node_map[i] =
+ (rte_node_t *)(node_map + stages) + nodes_per_stage * i;
+ for (j = 0; j < nodes_per_stage; j++) {
+ total_percent = 0;
+ for (k = 0; k < nodes_per_stage; k++)
+ total_percent += edge_map[i][j][k];
+ if (!total_percent)
+ continue;
+ node_patterns[graph_data->nb_nodes] =
+ malloc(RTE_NODE_NAMESIZE);
+ if (node_patterns[graph_data->nb_nodes] == NULL) {
+ printf("Failed to create memory for pattern\n");
+ goto pattern_name_free;
+ }
+
+ /* Clone a worker node */
+ snprintf(nname, sizeof(nname), "%d-%d", i, j);
+ node_map[i][j] =
+ graph_node_get(TEST_GRAPH_WRK_NAME, nname);
+ if (node_map[i][j] == RTE_NODE_ID_INVALID) {
+ printf("Failed to create node[%s]\n", nname);
+ graph_data->nb_nodes++;
+ goto pattern_name_free;
+ }
+ snprintf(node_patterns[graph_data->nb_nodes],
+ RTE_NODE_NAMESIZE, "%s",
+ rte_node_id_to_name(node_map[i][j]));
+ node_data =
+ &graph_data->node_data[graph_data->nb_nodes];
+ node_data->node_id = node_map[i][j];
+ node_data->is_sink = false;
+ graph_data->nb_nodes++;
+ }
+ }
+
+ for (i = 0; i < stages - 1; i++) {
+ for (j = 0; j < nodes_per_stage; j++) {
+ /* Count edges i.e connections of worker node to next */
+ node_data =
+ graph_get_node_data(graph_data, node_map[i][j]);
+ edges = graph_node_count_edges(i, j, nodes_per_stage,
+ edge_map, ename,
+ node_data, node_map);
+ if (edges == RTE_EDGE_ID_INVALID) {
+ printf("Invalid edge configuration\n");
+ goto pattern_name_free;
+ }
+ if (!edges)
+ continue;
+
+ /* Connect a node in stage 'i' to nodes
+ * in stage 'i + 1' with edges.
+ */
+ count = rte_node_edge_update(
+ node_map[i][j], 0,
+ (const char **)(uintptr_t)ename, edges);
+ for (k = 0; k < edges; k++)
+ free(ename[k]);
+ if (count != edges) {
+ printf("Couldn't add edges %d %d\n", edges,
+ count);
+ goto pattern_name_free;
+ }
+ }
+ }
+
+ /* Setup Source nodes */
+ for (i = 0; i < nb_srcs; i++) {
+ edges = 0;
+ total_percent = 0;
+ node_patterns[graph_data->nb_nodes] = malloc(RTE_NODE_NAMESIZE);
+ if (node_patterns[graph_data->nb_nodes] == NULL) {
+ printf("Failed to create memory for pattern\n");
+ goto pattern_name_free;
+ }
+ /* Clone a source node */
+ snprintf(nname, sizeof(nname), "%d", i);
+ src_nodes[i] =
+ graph_node_get(burst_one ? TEST_GRAPH_SRC_BRST_ONE_NAME
+ : TEST_GRAPH_SRC_NAME,
+ nname);
+ if (src_nodes[i] == RTE_NODE_ID_INVALID) {
+ printf("Failed to create node[%s]\n", nname);
+ graph_data->nb_nodes++;
+ goto pattern_name_free;
+ }
+ snprintf(node_patterns[graph_data->nb_nodes], RTE_NODE_NAMESIZE,
+ "%s", rte_node_id_to_name(src_nodes[i]));
+ node_data = &graph_data->node_data[graph_data->nb_nodes];
+ node_data->node_id = src_nodes[i];
+ node_data->is_sink = false;
+ graph_data->nb_nodes++;
+
+ /* Prepare next node list to connect to */
+ for (j = 0; j < nodes_per_stage; j++) {
+ if (!src_map[i][j])
+ continue;
+ ename[edges] = malloc(sizeof(char) * RTE_NODE_NAMESIZE);
+ snprintf(ename[edges], RTE_NODE_NAMESIZE, "%s",
+ rte_node_id_to_name(node_map[0][j]));
+ node_data->next_nodes[edges] = node_map[0][j];
+ node_data->next_percentage[edges] = src_map[i][j];
+ edges++;
+ total_percent += src_map[i][j];
+ }
+
+ if (!edges)
+ continue;
+ if (edges >= MAX_EDGES_PER_NODE || total_percent != 100) {
+ printf("Invalid edge configuration\n");
+ for (j = 0; j < edges; j++)
+ free(ename[j]);
+ goto pattern_name_free;
+ }
+
+ /* Connect to list of next nodes using edges */
+ count = rte_node_edge_update(src_nodes[i], 0,
+ (const char **)(uintptr_t)ename,
+ edges);
+ for (k = 0; k < edges; k++)
+ free(ename[k]);
+ if (count != edges) {
+ printf("Couldn't add edges %d %d\n", edges, count);
+ goto pattern_name_free;
+ }
+ }
+
+ /* Setup Sink nodes */
+ for (i = 0; i < nb_sinks; i++) {
+ node_patterns[graph_data->nb_nodes] = malloc(RTE_NODE_NAMESIZE);
+ if (node_patterns[graph_data->nb_nodes] == NULL) {
+ printf("Failed to create memory for pattern\n");
+ goto pattern_name_free;
+ }
+
+ /* Clone a sink node */
+ snprintf(nname, sizeof(nname), "%d", i);
+ snk_nodes[i] = graph_node_get(TEST_GRAPH_SNK_NAME, nname);
+ if (snk_nodes[i] == RTE_NODE_ID_INVALID) {
+ printf("Failed to create node[%s]\n", nname);
+ graph_data->nb_nodes++;
+ goto pattern_name_free;
+ }
+ snprintf(node_patterns[graph_data->nb_nodes], RTE_NODE_NAMESIZE,
+ "%s", rte_node_id_to_name(snk_nodes[i]));
+ node_data = &graph_data->node_data[graph_data->nb_nodes];
+ node_data->node_id = snk_nodes[i];
+ node_data->is_sink = true;
+ graph_data->nb_nodes++;
+ }
+
+ /* Connect last stage worker nodes to sink nodes */
+ for (i = 0; i < nodes_per_stage; i++) {
+ edges = 0;
+ total_percent = 0;
+ node_data = graph_get_node_data(graph_data,
+ node_map[stages - 1][i]);
+ /* Prepare list of sink nodes to connect to */
+ for (j = 0; j < nb_sinks; j++) {
+ if (!snk_map[i][j])
+ continue;
+ ename[edges] = malloc(sizeof(char) * RTE_NODE_NAMESIZE);
+ snprintf(ename[edges], RTE_NODE_NAMESIZE, "%s",
+ rte_node_id_to_name(snk_nodes[j]));
+ node_data->next_nodes[edges] = snk_nodes[j];
+ node_data->next_percentage[edges] = snk_map[i][j];
+ edges++;
+ total_percent += snk_map[i][j];
+ }
+ if (!edges)
+ continue;
+ if (edges >= MAX_EDGES_PER_NODE || total_percent != 100) {
+ printf("Invalid edge configuration\n");
+ for (j = 0; j < edges; j++)
+ free(ename[i]);
+ goto pattern_name_free;
+ }
+
+ /* Connect a worker node to a list of sink nodes */
+ count = rte_node_edge_update(node_map[stages - 1][i], 0,
+ (const char **)(uintptr_t)ename,
+ edges);
+ for (k = 0; k < edges; k++)
+ free(ename[k]);
+ if (count != edges) {
+ printf("Couldn't add edges %d %d\n", edges, count);
+ goto pattern_name_free;
+ }
+ }
+
+ /* Create a Graph */
+ gconf.socket_id = SOCKET_ID_ANY;
+ gconf.nb_node_patterns = graph_data->nb_nodes;
+ gconf.node_patterns = (const char **)(uintptr_t)node_patterns;
+
+ graph_id = rte_graph_create(gname, &gconf);
+ if (graph_id == RTE_GRAPH_ID_INVALID) {
+ printf("Graph creation failed with error = %d\n", rte_errno);
+ goto pattern_name_free;
+ }
+ graph_data->graph_id = graph_id;
+
+ for (i = 0; i < graph_data->nb_nodes; i++)
+ free(node_patterns[i]);
+ free(snk_nodes);
+ free(src_nodes);
+ free(node_patterns);
+ return 0;
+
+pattern_name_free:
+ for (i = 0; i < graph_data->nb_nodes; i++)
+ free(node_patterns[i]);
+snk_free:
+ free(snk_nodes);
+src_free:
+ free(src_nodes);
+pattern_free:
+ free(node_patterns);
+data_free:
+ free(graph_data->node_data);
+memzone_free:
+ rte_memzone_free(mz);
+ return -ENOMEM;
+}
+
+/* Worker thread function */
+static int
+_graph_perf_wrapper(void *args)
+{
+ struct graph_lcore_data *data = args;
+ struct rte_graph *graph;
+
+ /* Lookup graph */
+ graph = rte_graph_lookup(rte_graph_id_to_name(data->graph_id));
+
+ /* Graph walk until done */
+ while (!data->done)
+ rte_graph_walk(graph);
+
+ return 0;
+}
+
+static int
+measure_perf_get(rte_graph_t graph_id)
+{
+ const char *pattern = rte_graph_id_to_name(graph_id);
+ uint32_t lcore_id = rte_get_next_lcore(-1, 1, 0);
+ struct rte_graph_cluster_stats_param param;
+ struct rte_graph_cluster_stats *stats;
+ struct graph_lcore_data *data;
+
+ data = rte_zmalloc("Graph_perf", sizeof(struct graph_lcore_data),
+ RTE_CACHE_LINE_SIZE);
+ data->graph_id = graph_id;
+ data->done = 0;
+
+ /* Run graph worker thread function */
+ rte_eal_remote_launch(_graph_perf_wrapper, data, lcore_id);
+
+ /* Collect stats for few msecs */
+ if (rte_graph_has_stats_feature()) {
+ memset(¶m, 0, sizeof(param));
+ param.f = stdout;
+ param.socket_id = SOCKET_ID_ANY;
+ param.graph_patterns = &pattern;
+ param.nb_graph_patterns = 1;
+
+ stats = rte_graph_cluster_stats_create(¶m);
+ if (stats == NULL) {
+ printf("Failed to create stats\n");
+ return -ENOMEM;
+ }
+
+ rte_delay_ms(3E2);
+ rte_graph_cluster_stats_get(stats, true);
+ rte_delay_ms(1E3);
+ rte_graph_cluster_stats_get(stats, false);
+ rte_graph_cluster_stats_destroy(stats);
+ } else
+ rte_delay_ms(1E3);
+
+ data->done = 1;
+ rte_eal_wait_lcore(lcore_id);
+
+ return 0;
+}
+
+static inline void
+graph_fini(void)
+{
+ const struct rte_memzone *mz = rte_memzone_lookup(TEST_GRAPH_PERF_MZ);
+ struct test_graph_perf *graph_data;
+
+ if (mz == NULL)
+ return;
+ graph_data = mz->addr;
+
+ rte_graph_destroy(rte_graph_id_to_name(graph_data->graph_id));
+ free(graph_data->node_data);
+ rte_memzone_free(rte_memzone_lookup(TEST_GRAPH_PERF_MZ));
+}
+
+static int
+measure_perf(void)
+{
+ const struct rte_memzone *mz;
+ struct test_graph_perf *graph_data;
+
+ mz = rte_memzone_lookup(TEST_GRAPH_PERF_MZ);
+ graph_data = mz->addr;
+
+ return measure_perf_get(graph_data->graph_id);
+}
+
+static inline int
+graph_hr_4s_1n_1src_1snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_hr_4s_1n_1src_1snk_brst_one(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_hr_4s_1n_2src_1snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_hr_4s_1n_1src_2snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_tree_4s_4n_1src_4snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_reverse_tree_3s_4n_1src_1snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_parallel_tree_5s_4n_4src_4snk(void)
+{
+ return measure_perf();
+}
+
+/* Graph Topology
+ * nodes per stage: 1
+ * stages: 4
+ * src: 1
+ * sink: 1
+ */
+static inline int
+graph_init_hr(void)
+{
+ uint8_t edge_map[][1][1] = {
+ { {100} },
+ { {100} },
+ { {100} },
+ { {100} },
+ };
+ uint8_t src_map[][1] = { {100} };
+ uint8_t snk_map[][1] = { {100} };
+
+ return graph_init("graph_hr", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 1
+ * stages: 4
+ * src: 1
+ * sink: 1
+ */
+static inline int
+graph_init_hr_brst_one(void)
+{
+ uint8_t edge_map[][1][1] = {
+ { {100} },
+ { {100} },
+ { {100} },
+ { {100} },
+ };
+ uint8_t src_map[][1] = { {100} };
+ uint8_t snk_map[][1] = { {100} };
+
+ return graph_init("graph_hr", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 1);
+}
+
+/* Graph Topology
+ * nodes per stage: 1
+ * stages: 4
+ * src: 2
+ * sink: 1
+ */
+static inline int
+graph_init_hr_multi_src(void)
+{
+ uint8_t edge_map[][1][1] = {
+ { {100} },
+ { {100} },
+ { {100} },
+ { {100} },
+ };
+ uint8_t src_map[][1] = {
+ {100}, {100}
+ };
+ uint8_t snk_map[][1] = { {100} };
+
+ return graph_init("graph_hr", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 1
+ * stages: 4
+ * src: 1
+ * sink: 2
+ */
+static inline int
+graph_init_hr_multi_snk(void)
+{
+ uint8_t edge_map[][1][1] = {
+ { {100} },
+ { {100} },
+ { {100} },
+ { {100} },
+ };
+ uint8_t src_map[][1] = { {100} };
+ uint8_t snk_map[][2] = { {50, 50} };
+
+ return graph_init("graph_hr", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 4
+ * stages: 4
+ * src: 1
+ * sink: 4
+ */
+static inline int
+graph_init_tree(void)
+{
+ uint8_t edge_map[][4][4] = {
+ {
+ {100, 0, 0, 0},
+ {0, 0, 0, 0},
+ {0, 0, 0, 0},
+ {0, 0, 0, 0}
+ },
+ {
+ {50, 0, 0, 0},
+ {50, 0, 0, 0},
+ {0, 0, 0, 0},
+ {0, 0, 0, 0}
+ },
+ {
+ {33, 33, 0, 0},
+ {34, 34, 0, 0},
+ {33, 33, 0, 0},
+ {0, 0, 0, 0}
+ },
+ {
+ {25, 25, 25, 0},
+ {25, 25, 25, 0},
+ {25, 25, 25, 0},
+ {25, 25, 25, 0}
+ }
+ };
+ uint8_t src_map[][4] = { {100, 0, 0, 0} };
+ uint8_t snk_map[][4] = {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ };
+
+ return graph_init("graph_full_split", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 4
+ * stages: 3
+ * src: 1
+ * sink: 1
+ */
+static inline int
+graph_init_reverse_tree(void)
+{
+ uint8_t edge_map[][4][4] = {
+ {
+ {25, 25, 25, 25},
+ {25, 25, 25, 25},
+ {25, 25, 25, 25},
+ {25, 25, 25, 25}
+ },
+ {
+ {33, 33, 33, 33},
+ {33, 33, 33, 33},
+ {34, 34, 34, 34},
+ {0, 0, 0, 0}
+ },
+ {
+ {50, 50, 50, 0},
+ {50, 50, 50, 0},
+ {0, 0, 0, 0},
+ {0, 0, 0, 0}
+ },
+ };
+ uint8_t src_map[][4] = { {25, 25, 25, 25} };
+ uint8_t snk_map[][1] = { {100}, {100}, {0}, {0} };
+
+ return graph_init("graph_full_split", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 4
+ * stages: 5
+ * src: 4
+ * sink: 4
+ */
+static inline int
+graph_init_parallel_tree(void)
+{
+ uint8_t edge_map[][4][4] = {
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ };
+ uint8_t src_map[][4] = {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ };
+ uint8_t snk_map[][4] = {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ };
+
+ return graph_init("graph_parallel", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/** Graph Creation cheat sheet
+ * edge_map -> dictates graph flow from worker stage 0 to worker stage n-1.
+ * src_map -> dictates source nodes enqueue percentage to worker stage 0.
+ * snk_map -> dictates stage n-1 enqueue percentage to sink.
+ *
+ * Layout:
+ * edge_map[<nb_stages>][<nodes_per_stg>][<nodes_in_nxt_stg = nodes_per_stg>]
+ * src_map[<nb_sources>][<nodes_in_stage0 = nodes_per_stage>]
+ * snk_map[<nodes_in_stage(n-1) = nodes_per_stage>][<nb_sinks>]
+ *
+ * The last array dictates the percentage of received objs to enqueue to next
+ * stage.
+ *
+ * Note: edge_map[][0][] will always be unused as it will receive from source
+ *
+ * Example:
+ * Graph:
+ * http://bit.ly/2PqbqOy
+ * Each stage(n) connects to all nodes in the next stage in decreasing
+ * order.
+ * Since we can't resize the edge_map dynamically we get away by creating
+ * dummy nodes and assigning 0 percentages.
+ * Max nodes across all stages = 4
+ * stages = 3
+ * nb_src = 1
+ * nb_snk = 1
+ * // Stages
+ * edge_map[][4][4] = {
+ * // Nodes per stage
+ * {
+ * {25, 25, 25, 25},
+ * {25, 25, 25, 25},
+ * {25, 25, 25, 25},
+ * {25, 25, 25, 25}
+ * }, // This will be unused.
+ * {
+ * // Nodes enabled in current stage + prev stage enq %
+ * {33, 33, 33, 33},
+ * {33, 33, 33, 33},
+ * {34, 34, 34, 34},
+ * {0, 0, 0, 0}
+ * },
+ * {
+ * {50, 50, 50, 0},
+ * {50, 50, 50, 0},
+ * {0, 0, 0, 0},
+ * {0, 0, 0, 0}
+ * },
+ * };
+ * Above, each stage tells how much it should receive from previous except
+ * from stage_0.
+ *
+ * src_map[][4] = { {25, 25, 25, 25} };
+ * Here, we tell each source the % it has to send to stage_0 nodes. In
+ * case we want 2 source node we can declare as
+ * src_map[][4] = { {25, 25, 25, 25}, {25, 25, 25, 25} };
+ *
+ * snk_map[][1] = { {100}, {100}, {0}, {0} }
+ * Here, we tell stage - 1 nodes how much to enqueue to sink_0.
+ * If we have 2 sinks we can do as follows
+ * snk_map[][2] = { {50, 50}, {50, 50}, {0, 0}, {0, 0} }
+ */
+
+static struct unit_test_suite graph_perf_testsuite = {
+ .suite_name = "Graph library performance test suite",
+ .setup = graph_perf_setup,
+ .teardown = graph_perf_teardown,
+ .unit_test_cases = {
+ TEST_CASE_ST(graph_init_hr, graph_fini,
+ graph_hr_4s_1n_1src_1snk),
+ TEST_CASE_ST(graph_init_hr_brst_one, graph_fini,
+ graph_hr_4s_1n_1src_1snk_brst_one),
+ TEST_CASE_ST(graph_init_hr_multi_src, graph_fini,
+ graph_hr_4s_1n_2src_1snk),
+ TEST_CASE_ST(graph_init_hr_multi_snk, graph_fini,
+ graph_hr_4s_1n_1src_2snk),
+ TEST_CASE_ST(graph_init_tree, graph_fini,
+ graph_tree_4s_4n_1src_4snk),
+ TEST_CASE_ST(graph_init_reverse_tree, graph_fini,
+ graph_reverse_tree_3s_4n_1src_1snk),
+ TEST_CASE_ST(graph_init_parallel_tree, graph_fini,
+ graph_parallel_tree_5s_4n_4src_4snk),
+ TEST_CASES_END(), /**< NULL terminate unit test array */
+ },
+};
+
+static int
+test_graph_perf_func(void)
+{
+ return unit_test_suite_runner(&graph_perf_testsuite);
+}
+
+REGISTER_TEST_COMMAND(graph_perf_autotest, test_graph_perf_func);
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 15/26] node: add log infra and null node
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (13 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 14/26] graph: add performance testcase jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 16/26] node: add ethdev Rx node jerinj
` (11 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic,
Nithin Dabilpuram, Pavan Nikhilesh, Bruce Richardson
Cc: dev, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add log infra for node specific logging.
Also, add null rte_node that just ignores all the objects
directed to it.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
MAINTAINERS | 5 +++++
config/common_base | 5 +++++
doc/api/doxy-api.conf.in | 1 +
lib/Makefile | 3 +++
lib/librte_node/Makefile | 20 ++++++++++++++++++++
lib/librte_node/log.c | 14 ++++++++++++++
lib/librte_node/meson.build | 6 ++++++
lib/librte_node/node_private.h | 22 ++++++++++++++++++++++
lib/librte_node/null.c | 23 +++++++++++++++++++++++
lib/librte_node/rte_node_version.map | 6 ++++++
lib/meson.build | 5 ++++-
meson.build | 1 +
mk/rte.app.mk | 1 +
13 files changed, 111 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_node/Makefile
create mode 100644 lib/librte_node/log.c
create mode 100644 lib/librte_node/meson.build
create mode 100644 lib/librte_node/node_private.h
create mode 100644 lib/librte_node/null.c
create mode 100644 lib/librte_node/rte_node_version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index 32d0ea032..3959ed19a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1473,6 +1473,11 @@ M: Jerin Jacob <jerinj@marvell.com>
M: Kiran Kumar K <kirankumark@marvell.com>
F: lib/librte_graph/
+Nodes - EXPERIMENTAL
+M: Nithin Dabilpuram <ndabilpuram@marvell.com>
+M: Pavan Nikhilesh <pbhagavatula@marvell.com>
+F: lib/librte_node/
+
Test Applications
-----------------
diff --git a/config/common_base b/config/common_base
index 04a96aef5..442949ff1 100644
--- a/config/common_base
+++ b/config/common_base
@@ -1082,6 +1082,11 @@ CONFIG_RTE_LIBRTE_GRAPH=y
CONFIG_RTE_GRAPH_BURST_SIZE=256
CONFIG_RTE_LIBRTE_GRAPH_STATS=y
+#
+# Compile librte_node
+#
+CONFIG_RTE_LIBRTE_NODE=y
+
#
# Compile the test application
#
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 759a7213e..1d4f1a37d 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -49,6 +49,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/lib/librte_mempool \
@TOPDIR@/lib/librte_meter \
@TOPDIR@/lib/librte_metrics \
+ @TOPDIR@/lib/librte_node \
@TOPDIR@/lib/librte_net \
@TOPDIR@/lib/librte_pci \
@TOPDIR@/lib/librte_pdump \
diff --git a/lib/Makefile b/lib/Makefile
index 1f572b659..50d61a338 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -122,6 +122,9 @@ DEPDIRS-librte_rcu := librte_eal
DIRS-$(CONFIG_RTE_LIBRTE_GRAPH) += librte_graph
DEPDIRS-librte_graph := librte_eal
+DIRS-$(CONFIG_RTE_LIBRTE_NODE) += librte_node
+DEPDIRS-librte_node := librte_graph librte_lpm librte_ethdev librte_mbuf
+
ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
endif
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
new file mode 100644
index 000000000..5f9be6c2e
--- /dev/null
+++ b/lib/librte_node/Makefile
@@ -0,0 +1,20 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_node.a
+
+CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal -lrte_graph
+
+EXPORT_MAP := rte_node_version.map
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/log.c b/lib/librte_node/log.c
new file mode 100644
index 000000000..f035f91e8
--- /dev/null
+++ b/lib/librte_node/log.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include "node_private.h"
+
+int rte_node_logtype;
+
+RTE_INIT(rte_node_init_log)
+{
+ rte_node_logtype = rte_log_register("lib.node");
+ if (rte_node_logtype >= 0)
+ rte_log_set_level(rte_node_logtype, RTE_LOG_INFO);
+}
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
new file mode 100644
index 000000000..655970614
--- /dev/null
+++ b/lib/librte_node/meson.build
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+sources = files('null.c', 'log.c')
+allow_experimental_apis = true
+deps += ['graph']
diff --git a/lib/librte_node/node_private.h b/lib/librte_node/node_private.h
new file mode 100644
index 000000000..f30902a94
--- /dev/null
+++ b/lib/librte_node/node_private.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __NODE_PRIVATE_H__
+#define __NODE_PRIVATE_H__
+
+#include <rte_common.h>
+#include <rte_log.h>
+
+extern int rte_node_logtype;
+#define NODE_LOG(level, node_name, ...) \
+ rte_log(RTE_LOG_##level, rte_node_logtype, \
+ RTE_FMT("NODE %s: %s():%u " RTE_FMT_HEAD(__VA_ARGS__, ) "\n", \
+ node_name, __func__, __LINE__, \
+ RTE_FMT_TAIL(__VA_ARGS__, )))
+
+#define node_err(node_name, ...) NODE_LOG(ERR, node_name, __VA_ARGS__)
+#define node_info(node_name, ...) NODE_LOG(INFO, node_name, __VA_ARGS__)
+#define node_dbg(node_name, ...) NODE_LOG(DEBUG, node_name, __VA_ARGS__)
+
+#endif /* __NODE_PRIVATE_H__ */
diff --git a/lib/librte_node/null.c b/lib/librte_node/null.c
new file mode 100644
index 000000000..c7cd8b6df
--- /dev/null
+++ b/lib/librte_node/null.c
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_graph.h>
+
+static uint16_t
+null(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ RTE_SET_USED(node);
+ RTE_SET_USED(objs);
+ RTE_SET_USED(graph);
+
+ return nb_objs;
+}
+
+static struct rte_node_register null_node = {
+ .name = "null",
+ .process = null,
+};
+
+RTE_NODE_REGISTER(null_node);
diff --git a/lib/librte_node/rte_node_version.map b/lib/librte_node/rte_node_version.map
new file mode 100644
index 000000000..f87163bb9
--- /dev/null
+++ b/lib/librte_node/rte_node_version.map
@@ -0,0 +1,6 @@
+EXPERIMENTAL {
+ global:
+
+ rte_node_logtype;
+ local: *;
+};
diff --git a/lib/meson.build b/lib/meson.build
index c43d86bb9..147129b0b 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -30,7 +30,7 @@ libraries = [
# add pkt framework libs which use other libs from above
'port', 'table', 'pipeline',
# flow_classify lib depends on pkt framework table lib
- 'flow_classify', 'bpf', 'graph', 'telemetry']
+ 'flow_classify', 'bpf', 'graph', 'node', 'telemetry']
if is_windows
libraries = ['kvargs','eal'] # only supported libraries for windows
@@ -186,6 +186,9 @@ foreach l:libraries
dpdk_libraries = [shared_lib] + dpdk_libraries
dpdk_static_libraries = [static_lib] + dpdk_static_libraries
+ if libname == 'rte_node'
+ dpdk_graph_nodes = [static_lib]
+ endif
endif # sources.length() > 0
set_variable('shared_rte_' + name, shared_dep)
diff --git a/meson.build b/meson.build
index b7ae9c8d9..811c96421 100644
--- a/meson.build
+++ b/meson.build
@@ -16,6 +16,7 @@ cc = meson.get_compiler('c')
dpdk_conf = configuration_data()
dpdk_libraries = []
dpdk_static_libraries = []
+dpdk_graph_nodes = []
dpdk_driver_classes = []
dpdk_drivers = []
dpdk_extra_ldflags = []
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index b1195f09a..68d7806a4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -99,6 +99,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
_LDLIBS-$(CONFIG_RTE_LIBRTE_RCU) += -lrte_rcu
_LDLIBS-$(CONFIG_RTE_LIBRTE_GRAPH) += -lrte_graph
+_LDLIBS-$(CONFIG_RTE_LIBRTE_NODE) += -lrte_node
ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_KNI) += -lrte_kni
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 16/26] node: add ethdev Rx node
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (14 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 15/26] node: add log infra and null node jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 17/26] node: add ethdev Tx node jerinj
` (10 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add source rte_node ethdev_rx process function and register
it. This node is a source node that will be called periodically
and when called, performs rte_eth_rx_burst() on a specific
(port, queue) pair and enqueue them as stream of objects to
next node.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/Makefile | 3 +-
lib/librte_node/ethdev_rx.c | 221 +++++++++++++++++++++++++++++++
lib/librte_node/ethdev_rx_priv.h | 81 +++++++++++
lib/librte_node/meson.build | 4 +-
4 files changed, 306 insertions(+), 3 deletions(-)
create mode 100644 lib/librte_node/ethdev_rx.c
create mode 100644 lib/librte_node/ethdev_rx_priv.h
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index 5f9be6c2e..5a5f589ba 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -9,12 +9,13 @@ LIB = librte_node.a
CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS)
-LDLIBS += -lrte_eal -lrte_graph
+LDLIBS += -lrte_eal -lrte_graph -lrte_ethdev
EXPORT_MAP := rte_node_version.map
# all source are stored in SRCS-y
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/ethdev_rx.c b/lib/librte_node/ethdev_rx.c
new file mode 100644
index 000000000..5cc736598
--- /dev/null
+++ b/lib/librte_node/ethdev_rx.c
@@ -0,0 +1,221 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_ether.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_mbuf.h>
+
+#include "ethdev_rx_priv.h"
+#include "node_private.h"
+
+static struct ethdev_rx_node_main ethdev_rx_main;
+
+static __rte_always_inline uint16_t
+ethdev_rx_node_process_inline(struct rte_graph *graph, struct rte_node *node,
+ uint16_t port, uint16_t queue)
+{
+ uint16_t count, next_index = ETHDEV_RX_NEXT_IP4_LOOKUP;
+
+ /* Get pkts from port */
+ count = rte_eth_rx_burst(port, queue, (struct rte_mbuf **)node->objs,
+ RTE_GRAPH_BURST_SIZE);
+
+ if (!count)
+ return 0;
+ node->idx = count;
+ /* Enqueue to next node */
+ rte_node_next_stream_move(graph, node, next_index);
+
+ return count;
+}
+
+static __rte_always_inline uint16_t
+ethdev_rx_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t cnt)
+{
+ ethdev_rx_node_ctx_t *ctx = (ethdev_rx_node_ctx_t *)node->ctx;
+ uint16_t n_pkts = 0;
+
+ RTE_SET_USED(objs);
+ RTE_SET_USED(cnt);
+
+ n_pkts = ethdev_rx_node_process_inline(graph, node, ctx->port_id,
+ ctx->queue_id);
+ return n_pkts;
+}
+
+static inline uint32_t
+l3_ptype(uint16_t etype, uint32_t ptype)
+{
+ ptype = ptype & ~RTE_PTYPE_L3_MASK;
+ if (etype == rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV4))
+ ptype |= RTE_PTYPE_L3_IPV4_EXT_UNKNOWN;
+ else if (etype == rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV6))
+ ptype |= RTE_PTYPE_L3_IPV6_EXT_UNKNOWN;
+ return ptype;
+}
+
+/* Callback for soft ptype parsing */
+static uint16_t
+eth_pkt_parse_cb(uint16_t port, uint16_t queue, struct rte_mbuf **mbufs,
+ uint16_t nb_pkts, uint16_t max_pkts, void *user_param)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3;
+ struct rte_ether_hdr *eth_hdr;
+ uint16_t etype, n_left;
+ struct rte_mbuf **pkts;
+
+ RTE_SET_USED(port);
+ RTE_SET_USED(queue);
+ RTE_SET_USED(max_pkts);
+ RTE_SET_USED(user_param);
+
+ pkts = mbufs;
+ n_left = nb_pkts;
+ while (n_left >= 12) {
+
+ /* Prefetch next-next mbufs */
+ rte_prefetch0(pkts[8]);
+ rte_prefetch0(pkts[9]);
+ rte_prefetch0(pkts[10]);
+ rte_prefetch0(pkts[11]);
+
+ /* Prefetch next mbuf data */
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[4], struct rte_ether_hdr *));
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[5], struct rte_ether_hdr *));
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[6], struct rte_ether_hdr *));
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[7], struct rte_ether_hdr *));
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+ pkts += 4;
+ n_left -= 4;
+
+ /* Extract ptype of mbuf0 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf0->packet_type = l3_ptype(etype, 0);
+
+ /* Extract ptype of mbuf1 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf1, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf1->packet_type = l3_ptype(etype, 0);
+
+ /* Extract ptype of mbuf2 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf2, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf2->packet_type = l3_ptype(etype, 0);
+
+ /* Extract ptype of mbuf3 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf3, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf3->packet_type = l3_ptype(etype, 0);
+ }
+
+ while (n_left > 0) {
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left -= 1;
+
+ /* Extract ptype of mbuf0 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf0->packet_type = l3_ptype(etype, 0);
+ }
+
+ return nb_pkts;
+}
+
+#define MAX_PTYPES 16
+static int
+ethdev_ptype_setup(uint16_t port, uint16_t queue)
+{
+ uint8_t l3_ipv4 = 0, l3_ipv6 = 0;
+ uint32_t ptypes[MAX_PTYPES];
+ int i, rc;
+
+ /* Check IPv4 & IPv6 ptype support */
+ rc = rte_eth_dev_get_supported_ptypes(port, RTE_PTYPE_L3_MASK, ptypes,
+ MAX_PTYPES);
+ for (i = 0; i < rc; i++) {
+ if (ptypes[i] & RTE_PTYPE_L3_IPV4)
+ l3_ipv4 = 1;
+ if (ptypes[i] & RTE_PTYPE_L3_IPV6)
+ l3_ipv6 = 1;
+ }
+
+ if (!l3_ipv4 || !l3_ipv6) {
+ node_info("ethdev_rx",
+ "Enabling ptype callback for required ptypes on port %u\n",
+ port);
+
+ if (!rte_eth_add_rx_callback(port, queue, eth_pkt_parse_cb,
+ NULL)) {
+ node_err("ethdev_rx",
+ "Failed to add rx ptype cb: port=%d, queue=%d\n",
+ port, queue);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int
+ethdev_rx_node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ ethdev_rx_node_ctx_t *ctx = (ethdev_rx_node_ctx_t *)node->ctx;
+ ethdev_rx_node_elem_t *elem = ethdev_rx_main.head;
+
+ RTE_SET_USED(graph);
+
+ while (elem) {
+ if (elem->nid == node->id) {
+ /* Update node specific context */
+ memcpy(ctx, &elem->ctx, sizeof(ethdev_rx_node_ctx_t));
+ break;
+ }
+ elem = elem->next;
+ }
+
+ RTE_VERIFY(elem != NULL);
+
+ /* Check and setup ptype */
+ return ethdev_ptype_setup(ctx->port_id, ctx->queue_id);
+}
+
+struct ethdev_rx_node_main *
+ethdev_rx_get_node_data_get(void)
+{
+ return ðdev_rx_main;
+}
+
+static struct rte_node_register ethdev_rx_node_base = {
+ .process = ethdev_rx_node_process,
+ .flags = RTE_NODE_SOURCE_F,
+ .name = "ethdev_rx",
+
+ .init = ethdev_rx_node_init,
+
+ .nb_edges = ETHDEV_RX_NEXT_MAX,
+ .next_nodes = {[ETHDEV_RX_NEXT_IP4_LOOKUP] = "ip4_lookup"},
+};
+
+struct rte_node_register *
+ethdev_rx_node_get(void)
+{
+ return ðdev_rx_node_base;
+}
+
+RTE_NODE_REGISTER(ethdev_rx_node_base);
diff --git a/lib/librte_node/ethdev_rx_priv.h b/lib/librte_node/ethdev_rx_priv.h
new file mode 100644
index 000000000..2d7195a36
--- /dev/null
+++ b/lib/librte_node/ethdev_rx_priv.h
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#ifndef __INCLUDE_ETHDEV_RX_PRIV_H__
+#define __INCLUDE_ETHDEV_RX_PRIV_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+struct ethdev_rx_node_elem;
+struct ethdev_rx_node_ctx;
+typedef struct ethdev_rx_node_elem ethdev_rx_node_elem_t;
+typedef struct ethdev_rx_node_ctx ethdev_rx_node_ctx_t;
+
+/**
+ * @internal
+ *
+ * Ethernet device Rx node context structure.
+ */
+struct ethdev_rx_node_ctx {
+ uint16_t port_id; /**< Port identifier of the Rx node. */
+ uint16_t queue_id; /**< Queue identifier of the Rx node. */
+};
+
+/**
+ * @internal
+ *
+ * Ethernet device Rx node list element structure.
+ */
+struct ethdev_rx_node_elem {
+ struct ethdev_rx_node_elem *next;
+ /**< Pointer to the next Rx node element. */
+ struct ethdev_rx_node_ctx ctx;
+ /**< Rx node context. */
+ rte_node_t nid;
+ /**< Node identifier of the Rx node. */
+};
+
+enum ethdev_rx_next_nodes {
+ ETHDEV_RX_NEXT_IP4_LOOKUP,
+ ETHDEV_RX_NEXT_MAX,
+};
+
+/**
+ * @internal
+ *
+ * Ethernet Rx node main structure.
+ */
+struct ethdev_rx_node_main {
+ ethdev_rx_node_elem_t *head;
+ /**< Pointer to the head Rx node element. */
+};
+
+/**
+ * @internal
+ *
+ * Get the Ethernet Rx node data.
+ *
+ * @return
+ * Pointer to Ethernet Rx node data.
+ */
+struct ethdev_rx_node_main *ethdev_rx_get_node_data_get(void);
+
+/**
+ * @internal
+ *
+ * Get the Ethernet Rx node.
+ *
+ * @retrun
+ * Pointer to the Ethernet Rx node.
+ */
+struct rte_node_register *ethdev_rx_node_get(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_ETHDEV_RX_PRIV_H__ */
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index 655970614..4adfd6580 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -1,6 +1,6 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(C) 2020 Marvell International Ltd.
-sources = files('null.c', 'log.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c')
allow_experimental_apis = true
-deps += ['graph']
+deps += ['graph', 'ethdev']
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 17/26] node: add ethdev Tx node
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (15 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 16/26] node: add ethdev Rx node jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 18/26] node: add ethdev Rx and Tx node ctrl API jerinj
` (9 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add rte_node ethdev_tx process function and register it to
graph infra. This node has a specific (port, tx-queue) as context
and it enqueue's all the packets received to that specific queue pair.
When rte_eth_tx_burst() i.e enqueue to queue pair fails, packets
are forwarded to pkt_drop node to be free'd.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/Makefile | 3 +-
lib/librte_node/ethdev_tx.c | 86 ++++++++++++++++++++++++++++++++
lib/librte_node/ethdev_tx_priv.h | 62 +++++++++++++++++++++++
lib/librte_node/meson.build | 4 +-
4 files changed, 152 insertions(+), 3 deletions(-)
create mode 100644 lib/librte_node/ethdev_tx.c
create mode 100644 lib/librte_node/ethdev_tx_priv.h
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index 5a5f589ba..14a293982 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -9,7 +9,7 @@ LIB = librte_node.a
CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS)
-LDLIBS += -lrte_eal -lrte_graph -lrte_ethdev
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev
EXPORT_MAP := rte_node_version.map
@@ -17,5 +17,6 @@ EXPORT_MAP := rte_node_version.map
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/ethdev_tx.c b/lib/librte_node/ethdev_tx.c
new file mode 100644
index 000000000..075149089
--- /dev/null
+++ b/lib/librte_node/ethdev_tx.c
@@ -0,0 +1,86 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_mbuf.h>
+
+#include "ethdev_tx_priv.h"
+
+static struct ethdev_tx_node_main ethdev_tx_main;
+
+static uint16_t
+ethdev_tx_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ ethdev_tx_node_ctx_t *ctx = (ethdev_tx_node_ctx_t *)node->ctx;
+ uint16_t port, queue;
+ uint16_t count;
+
+ /* Get Tx port id */
+ port = ctx->port;
+ queue = ctx->queue;
+
+ count = rte_eth_tx_burst(port, queue, (struct rte_mbuf **)objs,
+ nb_objs);
+
+ /* Redirect unsent pkts to drop node */
+ if (count != nb_objs) {
+ rte_node_enqueue(graph, node, ETHDEV_TX_NEXT_PKT_DROP,
+ &objs[count], nb_objs - count);
+ }
+
+ return count;
+}
+
+static int
+ethdev_tx_node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ ethdev_tx_node_ctx_t *ctx = (ethdev_tx_node_ctx_t *)node->ctx;
+ uint64_t port_id = RTE_MAX_ETHPORTS;
+ int i;
+
+ /* Find our port id */
+ for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
+ if (ethdev_tx_main.nodes[i] == node->id) {
+ port_id = i;
+ break;
+ }
+ }
+ RTE_VERIFY(port_id < RTE_MAX_ETHPORTS);
+
+ /* Update port and queue */
+ ctx->port = port_id;
+ ctx->queue = graph->id;
+
+ return 0;
+}
+
+struct ethdev_tx_node_main *
+ethdev_tx_node_data_get(void)
+{
+ return ðdev_tx_main;
+}
+
+static struct rte_node_register ethdev_tx_node_base = {
+ .process = ethdev_tx_node_process,
+ .name = "ethdev_tx",
+
+ .init = ethdev_tx_node_init,
+
+ .nb_edges = ETHDEV_TX_NEXT_MAX,
+ .next_nodes = {
+ [ETHDEV_TX_NEXT_PKT_DROP] = "pkt_drop",
+ },
+};
+
+struct rte_node_register *
+ethdev_tx_node_get(void)
+{
+ return ðdev_tx_node_base;
+}
+
+RTE_NODE_REGISTER(ethdev_tx_node_base);
diff --git a/lib/librte_node/ethdev_tx_priv.h b/lib/librte_node/ethdev_tx_priv.h
new file mode 100644
index 000000000..586bff44a
--- /dev/null
+++ b/lib/librte_node/ethdev_tx_priv.h
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#ifndef __INCLUDE_ETHDEV_TX_PRIV_H__
+#define __INCLUDE_ETHDEV_TX_PRIV_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+struct ethdev_tx_node_ctx;
+typedef struct ethdev_tx_node_ctx ethdev_tx_node_ctx_t;
+
+enum ethdev_tx_next_nodes {
+ ETHDEV_TX_NEXT_PKT_DROP,
+ ETHDEV_TX_NEXT_MAX,
+};
+
+/**
+ * @internal
+ *
+ * Ethernet Tx node context structure.
+ */
+struct ethdev_tx_node_ctx {
+ uint16_t port; /**< Port identifier of the Ethernet Tx node. */
+ uint16_t queue; /**< Queue identifier of the Ethernet Tx node. */
+};
+
+/**
+ * @internal
+ *
+ * Ethernet Tx node main structure.
+ */
+struct ethdev_tx_node_main {
+ uint32_t nodes[RTE_MAX_ETHPORTS]; /**< Tx nodes for each ethdev port. */
+};
+
+/**
+ * @internal
+ *
+ * Get the Ethernet Tx node data.
+ *
+ * @return
+ * Pointer to Ethernet Tx node data.
+ */
+struct ethdev_tx_node_main *ethdev_tx_node_data_get(void);
+
+/**
+ * @internal
+ *
+ * Get the Ethernet Tx node.
+ *
+ * @retrun
+ * Pointer to the Ethernet Tx node.
+ */
+struct rte_node_register *ethdev_tx_node_get(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_ETHDEV_TX_PRIV_H__ */
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index 4adfd6580..c3e35276b 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -1,6 +1,6 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(C) 2020 Marvell International Ltd.
-sources = files('null.c', 'log.c', 'ethdev_rx.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c')
allow_experimental_apis = true
-deps += ['graph', 'ethdev']
+deps += ['graph', 'mbuf', 'ethdev']
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 18/26] node: add ethdev Rx and Tx node ctrl API
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (16 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 17/26] node: add ethdev Tx node jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 19/26] node: ipv4 lookup for arm64 jerinj
` (8 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: John McNamara, Marko Kovacevic, Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add ctrl api to setup ethdev_rx and ethdev_tx node.
This ctrl api clones 'N' number of ethdev_rx and ethdev_tx
nodes with specific (port, queue) pairs updated in their context.
All the ethdev ports and queues are setup before this api
is called.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
doc/api/doxy-api-index.md | 2 +
lib/librte_node/Makefile | 6 +-
lib/librte_node/ethdev_ctrl.c | 99 ++++++++++++++++++++++++++++
lib/librte_node/meson.build | 5 +-
lib/librte_node/node_private.h | 74 +++++++++++++++++++++
lib/librte_node/rte_node_eth_api.h | 70 ++++++++++++++++++++
lib/librte_node/rte_node_version.map | 1 +
7 files changed, 254 insertions(+), 3 deletions(-)
create mode 100644 lib/librte_node/ethdev_ctrl.c
create mode 100644 lib/librte_node/rte_node_eth_api.h
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index fd2ff64d7..1784674df 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -161,6 +161,8 @@ The public API headers are grouped by topics:
[table_action] (@ref rte_table_action.h)
* [graph] (@ref rte_graph.h):
[graph_worker] (@ref rte_graph_worker.h)
+ * graph_nodes:
+ [eth_node] (@ref rte_node_eth_api.h),
- **basic**:
[approx fraction] (@ref rte_approx.h),
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index 14a293982..c7b291c3c 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -9,7 +9,7 @@ LIB = librte_node.a
CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS)
-LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev -lrte_mempool
EXPORT_MAP := rte_node_version.map
@@ -18,5 +18,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
+
+# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_eth_api.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/ethdev_ctrl.c b/lib/librte_node/ethdev_ctrl.c
new file mode 100644
index 000000000..971cf0fb8
--- /dev/null
+++ b/lib/librte_node/ethdev_ctrl.c
@@ -0,0 +1,99 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_ether.h>
+#include <rte_graph.h>
+#include <rte_node_eth_api.h>
+
+#include "ethdev_rx_priv.h"
+#include "ethdev_tx_priv.h"
+#include "node_private.h"
+
+static struct ethdev_ctrl {
+ uint16_t nb_graphs;
+} ctrl;
+
+int
+rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
+ uint16_t nb_graphs)
+{
+ struct ethdev_tx_node_main *tx_node_data;
+ uint16_t tx_q_used, rx_q_used, port_id;
+ struct rte_node_register *tx_node;
+ char name[RTE_NODE_NAMESIZE];
+ struct rte_mempool *mp;
+ uint32_t id;
+ int i, j;
+
+ tx_node_data = ethdev_tx_node_data_get();
+ tx_node = ethdev_tx_node_get();
+ for (i = 0; i < nb_confs; i++) {
+ port_id = conf[i].port_id;
+
+ if (!rte_eth_dev_is_valid_port(port_id))
+ return -EINVAL;
+
+ /* Check for mbuf minimum private size requirement */
+ for (j = 0; j < conf[i].mp_count; j++) {
+ mp = conf[i].mp[j];
+ if (!mp)
+ continue;
+ /* Check for minimum private space */
+ if (rte_pktmbuf_priv_size(mp) <
+ RTE_NODE_MBUF_PRIV2_SIZE) {
+ node_err("ethdev",
+ "Minimum mbuf priv size requirement not met by mp %s",
+ mp->name);
+ return -EINVAL;
+ }
+ }
+
+ rx_q_used = conf[i].num_rx_queues;
+ tx_q_used = conf[i].num_tx_queues;
+ /* Check if we have a txq for each worker */
+ if (tx_q_used < nb_graphs)
+ return -EINVAL;
+
+ /* Create node for each rx port queue pair */
+ for (j = 0; j < rx_q_used; j++) {
+ struct ethdev_rx_node_main *rx_node_data;
+ struct rte_node_register *rx_node;
+ ethdev_rx_node_elem_t *elem;
+
+ rx_node_data = ethdev_rx_get_node_data_get();
+ rx_node = ethdev_rx_node_get();
+ snprintf(name, sizeof(name), "%u-%u", port_id, j);
+ /* Clone a new rx node with same edges as parent */
+ id = rte_node_clone(rx_node->id, name);
+ if (id == RTE_NODE_ID_INVALID)
+ return -EIO;
+
+ /* Add it to list of ethdev rx nodes for lookup */
+ elem = malloc(sizeof(ethdev_rx_node_elem_t));
+ memset(elem, 0, sizeof(ethdev_rx_node_elem_t));
+ elem->ctx.port_id = port_id;
+ elem->ctx.queue_id = j;
+ elem->nid = id;
+ elem->next = rx_node_data->head;
+ rx_node_data->head = elem;
+
+ node_dbg("ethdev", "Rx node %s-%s: is at %u",
+ rx_node->name, name, id);
+ }
+
+ /* Create a per port tx node from base node */
+ snprintf(name, sizeof(name), "%u", port_id);
+ /* Clone a new node with same edges as parent */
+ id = rte_node_clone(tx_node->id, name);
+ tx_node_data->nodes[port_id] = id;
+
+ node_dbg("ethdev", "Tx node %s-%s: is at %u", tx_node->name,
+ name, id);
+ }
+
+ ctrl.nb_graphs = nb_graphs;
+ return 0;
+}
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index c3e35276b..f34162aa7 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(C) 2020 Marvell International Ltd.
-sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ethdev_ctrl.c')
+headers = files('rte_node_eth_api.h')
allow_experimental_apis = true
-deps += ['graph', 'mbuf', 'ethdev']
+deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
diff --git a/lib/librte_node/node_private.h b/lib/librte_node/node_private.h
index f30902a94..141448546 100644
--- a/lib/librte_node/node_private.h
+++ b/lib/librte_node/node_private.h
@@ -6,7 +6,9 @@
#define __NODE_PRIVATE_H__
#include <rte_common.h>
+#include <rte_crypto.h>
#include <rte_log.h>
+#include <rte_mbuf.h>
extern int rte_node_logtype;
#define NODE_LOG(level, node_name, ...) \
@@ -19,4 +21,76 @@ extern int rte_node_logtype;
#define node_info(node_name, ...) NODE_LOG(INFO, node_name, __VA_ARGS__)
#define node_dbg(node_name, ...) NODE_LOG(DEBUG, node_name, __VA_ARGS__)
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node mbuf private data to store next hop, ttl and checksum.
+ */
+struct rte_node_mbuf_priv1 {
+ union {
+ /* IP4 rewrite */
+ struct {
+ uint16_t nh;
+ uint16_t ttl;
+ uint32_t cksum;
+ };
+
+ uint64_t u;
+ };
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node mbuf private data to store crypto operation.
+ */
+struct rte_node_mbuf_priv2 {
+ union {
+ /* Sym crypto */
+ struct {
+ struct rte_crypto_op op;
+ };
+ };
+} __rte_cache_aligned;
+
+#define RTE_NODE_MBUF_PRIV2_SIZE sizeof(struct rte_node_mbuf_priv2)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get mbuf_priv1 pointer from rte_mbuf.
+ *
+ * @param
+ * Pointer to the rte_mbuf.
+ *
+ * @return
+ * Pointer to the mbuf_priv1.
+ */
+static __rte_always_inline struct rte_node_mbuf_priv1 *
+rte_node_mbuf_priv1(struct rte_mbuf *m)
+{
+ return (struct rte_node_mbuf_priv1 *)&m->udata64;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get mbuf_priv2 pointer from rte_mbuf.
+ *
+ * @param
+ * Pointer to the rte_mbuf.
+ *
+ * @return
+ * Pointer to the mbuf_priv2.
+ */
+static __rte_always_inline struct rte_node_mbuf_priv2 *
+rte_node_mbuf_priv2(struct rte_mbuf *m)
+{
+ return (struct rte_node_mbuf_priv2 *)rte_mbuf_to_priv(m);
+}
+
#endif /* __NODE_PRIVATE_H__ */
diff --git a/lib/librte_node/rte_node_eth_api.h b/lib/librte_node/rte_node_eth_api.h
new file mode 100644
index 000000000..39b31b45b
--- /dev/null
+++ b/lib/librte_node/rte_node_eth_api.h
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __INCLUDE_RTE_NODE_ETH_API_H__
+#define __INCLUDE_RTE_NODE_ETH_API_H__
+
+/**
+ * @file rte_node_eth_api.h
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * This API allows to setup ethdev_rx and ethdev_tx nodes
+ * and its queue associations.
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+#include <rte_mempool.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Port config for ethdev_rx and ethdev_tx node.
+ */
+struct rte_node_ethdev_config {
+ uint16_t port_id;
+ /**< Port identifier */
+ uint16_t num_rx_queues;
+ /**< Number of Rx queues. */
+ uint16_t num_tx_queues;
+ /**< Number of Tx queues. */
+ struct rte_mempool **mp;
+ /**< Array of mempools associated to Rx queue. */
+ uint16_t mp_count;
+ /**< Size of mp array. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initializes ethdev nodes.
+ *
+ * @param cfg
+ * Array of ethdev config that identifies which port's
+ * ethdev_rx and ethdev_tx nodes need to be created
+ * and queue association.
+ * @param cnt
+ * Size of cfg array.
+ * @param nb_graphs
+ * Number of graphs that will be used.
+ *
+ * @return
+ * 0 on successful initialization, negative otherwise.
+ */
+__rte_experimental
+int rte_node_eth_config(struct rte_node_ethdev_config *cfg,
+ uint16_t cnt, uint16_t nb_graphs);
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_RTE_NODE_ETH_API_H__ */
diff --git a/lib/librte_node/rte_node_version.map b/lib/librte_node/rte_node_version.map
index f87163bb9..c6c71bd02 100644
--- a/lib/librte_node/rte_node_version.map
+++ b/lib/librte_node/rte_node_version.map
@@ -1,6 +1,7 @@
EXPERIMENTAL {
global:
+ rte_node_eth_config;
rte_node_logtype;
local: *;
};
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 19/26] node: ipv4 lookup for arm64
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (17 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 18/26] node: add ethdev Rx and Tx node ctrl API jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86 jerinj
` (7 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: John McNamara, Marko Kovacevic, Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add arm64 specific IPv4 lookup process function
for ip4_lookup node. This node performs LPM lookup
on every packet received and forwards it to a next
node that is identified by lookup result.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
doc/api/doxy-api-index.md | 1 +
lib/librte_node/Makefile | 4 +-
lib/librte_node/ip4_lookup.c | 306 +++++++++++++++++++++++++++++
lib/librte_node/meson.build | 5 +-
lib/librte_node/rte_node_ip4_api.h | 43 ++++
5 files changed, 356 insertions(+), 3 deletions(-)
create mode 100644 lib/librte_node/ip4_lookup.c
create mode 100644 lib/librte_node/rte_node_ip4_api.h
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 1784674df..3908fddde 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -163,6 +163,7 @@ The public API headers are grouped by topics:
[graph_worker] (@ref rte_graph_worker.h)
* graph_nodes:
[eth_node] (@ref rte_node_eth_api.h),
+ [ip4_node] (@ref rte_node_ip4_api.h)
- **basic**:
[approx fraction] (@ref rte_approx.h),
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index c7b291c3c..d85a9e9eb 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -9,7 +9,7 @@ LIB = librte_node.a
CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS)
-LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev -lrte_mempool
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_lpm -lrte_ethdev -lrte_mempool
EXPORT_MAP := rte_node_version.map
@@ -19,8 +19,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_eth_api.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c
new file mode 100644
index 000000000..d7fcd1158
--- /dev/null
+++ b/lib/librte_node/ip4_lookup.c
@@ -0,0 +1,306 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <arpa/inet.h>
+#include <sys/socket.h>
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_ether.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_ip.h>
+#include <rte_lpm.h>
+#include <rte_mbuf.h>
+#include <rte_node_ip4_api.h>
+#include <rte_tcp.h>
+#include <rte_udp.h>
+
+#include "node_private.h"
+
+#define IPV4_L3FWD_LPM_MAX_RULES 1024
+#define IPV4_L3FWD_LPM_NUMBER_TBL8S (1 << 8)
+
+/* IP4 Lookup global data struct */
+struct ip4_lookup_node_main {
+ struct rte_lpm *lpm_tbl[RTE_MAX_NUMA_NODES];
+};
+
+#if defined(RTE_MACHINE_CPUFLAG_NEON)
+/* ARM64 NEON */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+ struct rte_ipv4_hdr *ipv4_hdr;
+ struct rte_ether_hdr *eth_hdr;
+ void **to_next, **from;
+ uint16_t last_spec = 0;
+ rte_edge_t next_index;
+ uint16_t n_left_from;
+ struct rte_lpm *lpm;
+ uint16_t held = 0;
+ uint32_t drop_nh;
+ rte_xmm_t result;
+ rte_xmm_t priv01;
+ rte_xmm_t priv23;
+ int32x4_t dip;
+ int rc, i;
+
+ /* Speculative next */
+ next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+ /* Drop node */
+ drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
+
+ /* Get socket specific LPM from ctx */
+ lpm = *((struct rte_lpm **)node->ctx);
+
+ pkts = (struct rte_mbuf **)objs;
+ from = objs;
+ n_left_from = nb_objs;
+
+#define OBJS_PER_CLINE (RTE_CACHE_LINE_SIZE / sizeof(void *))
+ for (i = OBJS_PER_CLINE; i < RTE_GRAPH_BURST_SIZE; i += OBJS_PER_CLINE)
+ rte_prefetch0(&objs[i]);
+
+ for (i = 0; i < 4 && i < n_left_from; i++) {
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[i], struct rte_ether_hdr *) + 1);
+ }
+
+ /* Get stream for the speculated next node */
+ to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+ while (n_left_from >= 4) {
+#if RTE_GRAPH_BURST_SIZE > 64
+ /* Prefetch next-next mbufs */
+ if (likely(n_left_from >= 11)) {
+ rte_prefetch0(pkts[8]);
+ rte_prefetch0(pkts[9]);
+ rte_prefetch0(pkts[10]);
+ rte_prefetch0(pkts[11]);
+ }
+#endif
+ /* Prefetch next mbuf data */
+ if (likely(n_left_from >= 7)) {
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
+ struct rte_ether_hdr *) +
+ 1);
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
+ struct rte_ether_hdr *) +
+ 1);
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
+ struct rte_ether_hdr *) +
+ 1);
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
+ struct rte_ether_hdr *) +
+ 1);
+ }
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+
+ pkts += 4;
+ n_left_from -= 4;
+
+ /* Extract DIP of mbuf0 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 0);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv01.u16[1] = ipv4_hdr->time_to_live;
+ priv01.u32[1] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf1 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf1, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 1);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv01.u16[5] = ipv4_hdr->time_to_live;
+ priv01.u32[3] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf2 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf2, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 2);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv23.u16[1] = ipv4_hdr->time_to_live;
+ priv23.u32[1] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf3 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf3, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 3);
+
+ dip = vreinterpretq_s32_u8(
+ vrev32q_u8(vreinterpretq_u8_s32(dip)));
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv23.u16[5] = ipv4_hdr->time_to_live;
+ priv23.u32[3] = ipv4_hdr->hdr_checksum;
+
+ /* Perform LPM lookup to get NH and next node */
+ rte_lpm_lookupx4(lpm, dip, result.u32, drop_nh);
+ priv01.u16[0] = result.u16[0];
+ priv01.u16[4] = result.u16[2];
+ priv23.u16[0] = result.u16[4];
+ priv23.u16[4] = result.u16[6];
+
+ rte_node_mbuf_priv1(mbuf0)->u = priv01.u64[0];
+ rte_node_mbuf_priv1(mbuf1)->u = priv01.u64[1];
+ rte_node_mbuf_priv1(mbuf2)->u = priv23.u64[0];
+ rte_node_mbuf_priv1(mbuf3)->u = priv23.u64[1];
+
+ /* Enqueue four to next node */
+ rte_edge_t fix_spec = ((next_index == result.u16[1]) &&
+ (result.u16[1] == result.u16[3]) &&
+ (result.u16[3] == result.u16[5]) &&
+ (result.u16[5] == result.u16[7]));
+
+ if (unlikely(fix_spec == 0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ /* Next0 */
+ if (next_index == result.u16[1]) {
+ to_next[0] = from[0];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[1],
+ from[0]);
+ }
+
+ /* Next1 */
+ if (next_index == result.u16[3]) {
+ to_next[0] = from[1];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[3],
+ from[1]);
+ }
+
+ /* Next2 */
+ if (next_index == result.u16[5]) {
+ to_next[0] = from[2];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[5],
+ from[2]);
+ }
+
+ /* Next3 */
+ if (next_index == result.u16[7]) {
+ to_next[0] = from[3];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[7],
+ from[3]);
+ }
+
+ from += 4;
+ } else {
+ last_spec += 4;
+ }
+ }
+
+ while (n_left_from > 0) {
+ uint32_t next_hop;
+ uint16_t next0;
+
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left_from -= 1;
+
+ /* Extract DIP of mbuf0 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+ rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
+ &next_hop);
+ next_hop = (rc == 0) ? next_hop : drop_nh;
+
+ rte_node_mbuf_priv1(mbuf0)->nh = (uint16_t)next_hop;
+ next_hop = next_hop >> 16;
+ next0 = (uint16_t)next_hop;
+
+ if (unlikely(next_index ^ next0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ rte_node_enqueue_x1(graph, node, next0, from[0]);
+ from += 1;
+ } else {
+ last_spec += 1;
+ }
+ }
+
+ /* !!! Home run !!! */
+ if (likely(last_spec == nb_objs)) {
+ rte_node_next_stream_move(graph, node, next_index);
+ return nb_objs;
+ }
+ held += last_spec;
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ rte_node_next_stream_put(graph, node, next_index, held);
+
+ return nb_objs;
+}
+
+#else
+
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ RTE_SET_USED(graph);
+ RTE_SET_USED(node);
+ RTE_SET_USED(objs);
+ RTE_SET_USED(nb_objs);
+ return nb_objs;
+}
+
+#endif
+
+static int
+ip4_lookup_node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ RTE_SET_USED(graph);
+ RTE_SET_USED(node);
+
+ node_dbg("ip4_lookup", "Initialized ip4_lookup node");
+
+ return 0;
+}
+
+static struct rte_node_register ip4_lookup_node = {
+ .process = ip4_lookup_node_process,
+ .name = "ip4_lookup",
+
+ .init = ip4_lookup_node_init,
+
+ .nb_edges = RTE_NODE_IP4_LOOKUP_NEXT_MAX,
+ .next_nodes = {
+ [RTE_NODE_IP4_LOOKUP_NEXT_REWRITE] = "ip4_rewrite",
+ [RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP] = "pkt_drop",
+ },
+};
+
+RTE_NODE_REGISTER(ip4_lookup_node);
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index f34162aa7..ddea1237a 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -1,7 +1,8 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(C) 2020 Marvell International Ltd.
-sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ethdev_ctrl.c')
-headers = files('rte_node_eth_api.h')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
+ 'ethdev_ctrl.c')
+headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
allow_experimental_apis = true
deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
diff --git a/lib/librte_node/rte_node_ip4_api.h b/lib/librte_node/rte_node_ip4_api.h
new file mode 100644
index 000000000..37c12bf82
--- /dev/null
+++ b/lib/librte_node/rte_node_ip4_api.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __INCLUDE_RTE_NODE_IP4_API_H__
+#define __INCLUDE_RTE_NODE_IP4_API_H__
+
+/**
+ * @file rte_node_ip4_api.h
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * This API allows to do control path functions of ip4_* nodes
+ * like ip4_lookup, ip4_rewrite.
+ *
+ */
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * IP4 lookup next nodes.
+ */
+enum rte_node_ip4_lookup_next {
+ RTE_NODE_IP4_LOOKUP_NEXT_REWRITE,
+ /**< Rewrite node. */
+ RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP,
+ /**< Packet drop node. */
+ RTE_NODE_IP4_LOOKUP_NEXT_MAX,
+ /**< Number of next nodes of lookup node. */
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_RTE_NODE_IP4_API_H__ */
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (18 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 19/26] node: ipv4 lookup for arm64 jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-19 12:25 ` Ray Kinsella
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 21/26] node: add ipv4 rewrite node jerinj
` (6 subsequent siblings)
26 siblings, 1 reply; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Pavan Nikhilesh <pbhagavatula@marvell.com>
Add IPv4 lookup process function for ip4_lookup
rte_node. This node performs LPM lookup using x86_64
vector supported RTE_LPM API on every packet received
and forwards it to a next node that is identified by
lookup result.
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/ip4_lookup.c | 245 +++++++++++++++++++++++++++++++++++
1 file changed, 245 insertions(+)
diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c
index d7fcd1158..c003e9c91 100644
--- a/lib/librte_node/ip4_lookup.c
+++ b/lib/librte_node/ip4_lookup.c
@@ -264,6 +264,251 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
return nb_objs;
}
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+ rte_edge_t next0, next1, next2, next3, next_index;
+ struct rte_ipv4_hdr *ipv4_hdr;
+ struct rte_ether_hdr *eth_hdr;
+ uint32_t ip0, ip1, ip2, ip3;
+ void **to_next, **from;
+ uint16_t last_spec = 0;
+ uint16_t n_left_from;
+ struct rte_lpm *lpm;
+ uint16_t held = 0;
+ uint32_t drop_nh;
+ rte_xmm_t dst;
+ __m128i dip; /* SSE register */
+ int rc, i;
+
+ /* Speculative next */
+ next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+ /* Drop node */
+ drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
+
+ /* Get socket specific LPM from ctx */
+ lpm = *((struct rte_lpm **)node->ctx);
+
+ pkts = (struct rte_mbuf **)objs;
+ from = objs;
+ n_left_from = nb_objs;
+
+ if (n_left_from >= 4) {
+ for (i = 0; i < 4; i++) {
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
+ struct rte_ether_hdr *) +
+ 1);
+ }
+ }
+
+ /* Get stream for the speculated next node */
+ to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+ while (n_left_from >= 4) {
+ /* Prefetch next-next mbufs */
+ if (likely(n_left_from >= 11)) {
+ rte_prefetch0(pkts[8]);
+ rte_prefetch0(pkts[9]);
+ rte_prefetch0(pkts[10]);
+ rte_prefetch0(pkts[11]);
+ }
+
+ /* Prefetch next mbuf data */
+ if (likely(n_left_from >= 7)) {
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
+ struct rte_ether_hdr *) +
+ 1);
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
+ struct rte_ether_hdr *) +
+ 1);
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
+ struct rte_ether_hdr *) +
+ 1);
+ rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
+ struct rte_ether_hdr *) +
+ 1);
+ }
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+
+ pkts += 4;
+ n_left_from -= 4;
+
+ /* Extract DIP of mbuf0 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ ip0 = ipv4_hdr->dst_addr;
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+ /* Extract DIP of mbuf1 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf1, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ ip1 = ipv4_hdr->dst_addr;
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr->time_to_live;
+
+ /* Extract DIP of mbuf2 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf2, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ ip2 = ipv4_hdr->dst_addr;
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr->time_to_live;
+
+ /* Extract DIP of mbuf3 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf3, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ ip3 = ipv4_hdr->dst_addr;
+
+ /* Prepare for lookup x4 */
+ dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+ /* Byte swap 4 IPV4 addresses. */
+ const __m128i bswap_mask = _mm_set_epi8(
+ 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+ dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr->time_to_live;
+
+ /* Perform LPM lookup to get NH and next node */
+ rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+ /* Extract next node id and NH */
+ rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] & 0xFFFF;
+ next0 = (dst.u32[0] >> 16);
+
+ rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] & 0xFFFF;
+ next1 = (dst.u32[1] >> 16);
+
+ rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] & 0xFFFF;
+ next2 = (dst.u32[2] >> 16);
+
+ rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] & 0xFFFF;
+ next3 = (dst.u32[3] >> 16);
+
+ /* Enqueue four to next node */
+ rte_edge_t fix_spec =
+ (next_index ^ next0) | (next_index ^ next1) |
+ (next_index ^ next2) | (next_index ^ next3);
+
+ if (unlikely(fix_spec)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ /* Next0 */
+ if (next_index == next0) {
+ to_next[0] = from[0];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next0,
+ from[0]);
+ }
+
+ /* Next1 */
+ if (next_index == next1) {
+ to_next[0] = from[1];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next1,
+ from[1]);
+ }
+
+ /* Next2 */
+ if (next_index == next2) {
+ to_next[0] = from[2];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next2,
+ from[2]);
+ }
+
+ /* Next3 */
+ if (next_index == next3) {
+ to_next[0] = from[3];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next3,
+ from[3]);
+ }
+
+ from += 4;
+
+ } else {
+ last_spec += 4;
+ }
+ }
+
+ while (n_left_from > 0) {
+ uint32_t next_hop;
+
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left_from -= 1;
+
+ /* Extract DIP of mbuf0 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+ ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+ rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
+ &next_hop);
+ next_hop = (rc == 0) ? next_hop : drop_nh;
+
+ rte_node_mbuf_priv1(mbuf0)->nh = next_hop & 0xFFFF;
+ next0 = (next_hop >> 16);
+
+ if (unlikely(next_index ^ next0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ rte_node_enqueue_x1(graph, node, next0, from[0]);
+ from += 1;
+ } else {
+ last_spec += 1;
+ }
+ }
+
+ /* !!! Home run !!! */
+ if (likely(last_spec == nb_objs)) {
+ rte_node_next_stream_move(graph, node, next_index);
+ return nb_objs;
+ }
+
+ held += last_spec;
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ rte_node_next_stream_put(graph, node, next_index, held);
+
+ return nb_objs;
+}
+
#else
static uint16_t
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 21/26] node: add ipv4 rewrite node
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (19 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86 jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 22/26] node: add ipv4 rewrite and lookup ctrl API jerinj
` (5 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Kiran Kumar K <kirankumark@marvell.com>
Add ip4 rewrite process function for ip4_rewrite
rte_node. On every packet received by this node,
header is overwritten with new data before forwarding
it to next node. Header data to overwrite with is
identified by next hop id passed in mbuf priv data
by previous node.
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
lib/librte_node/Makefile | 1 +
lib/librte_node/ip4_rewrite.c | 269 +++++++++++++++++++++++++++++
lib/librte_node/ip4_rewrite_priv.h | 55 ++++++
lib/librte_node/meson.build | 2 +-
4 files changed, 326 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_node/ip4_rewrite.c
create mode 100644 lib/librte_node/ip4_rewrite_priv.h
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index d85a9e9eb..be7e7e338 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -20,6 +20,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_rewrite.c
# install header files
SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
diff --git a/lib/librte_node/ip4_rewrite.c b/lib/librte_node/ip4_rewrite.c
new file mode 100644
index 000000000..f317f7f4f
--- /dev/null
+++ b/lib/librte_node/ip4_rewrite.c
@@ -0,0 +1,269 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_ether.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_ip.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_node_ip4_api.h>
+#include <rte_tcp.h>
+#include <rte_udp.h>
+#include <rte_vect.h>
+
+#include "ip4_rewrite_priv.h"
+#include "node_private.h"
+
+static struct ip4_rewrite_node_main *ip4_rewrite_nm;
+
+static uint16_t
+ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+ struct ip4_rewrite_nh_header *nh = ip4_rewrite_nm->nh;
+ uint16_t next0, next1, next2, next3, next_index;
+ struct rte_ipv4_hdr *ip0, *ip1, *ip2, *ip3;
+ uint16_t n_left_from, held = 0, last_spec = 0;
+ void *d0, *d1, *d2, *d3;
+ void **to_next, **from;
+ rte_xmm_t priv01;
+ rte_xmm_t priv23;
+ int i;
+
+ /* Speculative next as last next */
+ next_index = *(uint16_t *)node->ctx;
+ rte_prefetch0(nh);
+
+ pkts = (struct rte_mbuf **)objs;
+ from = objs;
+ n_left_from = nb_objs;
+
+ for (i = 0; i < 4 && i < n_left_from; i++)
+ rte_prefetch0(pkts[i]);
+
+ /* Get stream for the speculated next node */
+ to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+ /* Update Ethernet header of pkts */
+ while (n_left_from >= 4) {
+ if (likely(n_left_from >= 7)) {
+ /* Prefetch only next-mbuf struct and priv area.
+ * Data need not be prefetched as we only write.
+ */
+ rte_prefetch0(pkts[4]);
+ rte_prefetch0(pkts[5]);
+ rte_prefetch0(pkts[6]);
+ rte_prefetch0(pkts[7]);
+ }
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+
+ pkts += 4;
+ n_left_from -= 4;
+ priv01.u64[0] = rte_node_mbuf_priv1(mbuf0)->u;
+ priv01.u64[1] = rte_node_mbuf_priv1(mbuf1)->u;
+ priv23.u64[0] = rte_node_mbuf_priv1(mbuf2)->u;
+ priv23.u64[1] = rte_node_mbuf_priv1(mbuf3)->u;
+
+ /* Increment checksum by one. */
+ priv01.u32[1] += rte_cpu_to_be_16(0x0100);
+ priv01.u32[3] += rte_cpu_to_be_16(0x0100);
+ priv23.u32[1] += rte_cpu_to_be_16(0x0100);
+ priv23.u32[3] += rte_cpu_to_be_16(0x0100);
+
+ /* Update ttl,cksum rewrite ethernet hdr on mbuf0 */
+ d0 = rte_pktmbuf_mtod(mbuf0, void *);
+ rte_memcpy(d0, nh[priv01.u16[0]].rewrite_data,
+ nh[priv01.u16[0]].rewrite_len);
+
+ next0 = nh[priv01.u16[0]].tx_node;
+ ip0 = (struct rte_ipv4_hdr *)((uint8_t *)d0 +
+ sizeof(struct rte_ether_hdr));
+ ip0->time_to_live = priv01.u16[1] - 1;
+ ip0->hdr_checksum = priv01.u16[2] + priv01.u16[3];
+
+ /* Update ttl,cksum rewrite ethernet hdr on mbuf1 */
+ d1 = rte_pktmbuf_mtod(mbuf1, void *);
+ rte_memcpy(d1, nh[priv01.u16[4]].rewrite_data,
+ nh[priv01.u16[4]].rewrite_len);
+
+ next1 = nh[priv01.u16[4]].tx_node;
+ ip1 = (struct rte_ipv4_hdr *)((uint8_t *)d1 +
+ sizeof(struct rte_ether_hdr));
+ ip1->time_to_live = priv01.u16[5] - 1;
+ ip1->hdr_checksum = priv01.u16[6] + priv01.u16[7];
+
+ /* Update ttl,cksum rewrite ethernet hdr on mbuf2 */
+ d2 = rte_pktmbuf_mtod(mbuf2, void *);
+ rte_memcpy(d2, nh[priv23.u16[0]].rewrite_data,
+ nh[priv23.u16[0]].rewrite_len);
+ next2 = nh[priv23.u16[0]].tx_node;
+ ip2 = (struct rte_ipv4_hdr *)((uint8_t *)d2 +
+ sizeof(struct rte_ether_hdr));
+ ip2->time_to_live = priv23.u16[1] - 1;
+ ip2->hdr_checksum = priv23.u16[2] + priv23.u16[3];
+
+ /* Update ttl,cksum rewrite ethernet hdr on mbuf3 */
+ d3 = rte_pktmbuf_mtod(mbuf3, void *);
+ rte_memcpy(d3, nh[priv23.u16[4]].rewrite_data,
+ nh[priv23.u16[4]].rewrite_len);
+
+ next3 = nh[priv23.u16[4]].tx_node;
+ ip3 = (struct rte_ipv4_hdr *)((uint8_t *)d3 +
+ sizeof(struct rte_ether_hdr));
+ ip3->time_to_live = priv23.u16[5] - 1;
+ ip3->hdr_checksum = priv23.u16[6] + priv23.u16[7];
+
+ /* Enqueue four to next node */
+ rte_edge_t fix_spec =
+ ((next_index == next0) && (next0 == next1) &&
+ (next1 == next2) && (next2 == next3));
+
+ if (unlikely(fix_spec == 0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ /* next0 */
+ if (next_index == next0) {
+ to_next[0] = from[0];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next0,
+ from[0]);
+ }
+
+ /* next1 */
+ if (next_index == next1) {
+ to_next[0] = from[1];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next1,
+ from[1]);
+ }
+
+ /* next2 */
+ if (next_index == next2) {
+ to_next[0] = from[2];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next2,
+ from[2]);
+ }
+
+ /* next3 */
+ if (next_index == next3) {
+ to_next[0] = from[3];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next3,
+ from[3]);
+ }
+
+ from += 4;
+
+ /* Change speculation if last two are same */
+ if ((next_index != next3) && (next2 == next3)) {
+ /* Put the current speculated node */
+ rte_node_next_stream_put(graph, node,
+ next_index, held);
+ held = 0;
+
+ /* Get next speculated stream */
+ next_index = next3;
+ to_next = rte_node_next_stream_get(
+ graph, node, next_index, nb_objs);
+ }
+ } else {
+ last_spec += 4;
+ }
+ }
+
+ while (n_left_from > 0) {
+ uint16_t chksum;
+
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left_from -= 1;
+
+ d0 = rte_pktmbuf_mtod(mbuf0, void *);
+ rte_memcpy(d0, nh[rte_node_mbuf_priv1(mbuf0)->nh].rewrite_data,
+ nh[rte_node_mbuf_priv1(mbuf0)->nh].rewrite_len);
+
+ next0 = nh[rte_node_mbuf_priv1(mbuf0)->nh].tx_node;
+ ip0 = (struct rte_ipv4_hdr *)((uint8_t *)d0 +
+ sizeof(struct rte_ether_hdr));
+ chksum = rte_node_mbuf_priv1(mbuf0)->cksum +
+ rte_cpu_to_be_16(0x0100);
+ chksum += chksum >= 0xffff;
+ ip0->hdr_checksum = chksum;
+ ip0->time_to_live = rte_node_mbuf_priv1(mbuf0)->ttl - 1;
+
+ if (unlikely(next_index ^ next0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ rte_node_enqueue_x1(graph, node, next0, from[0]);
+ from += 1;
+ } else {
+ last_spec += 1;
+ }
+ }
+
+ /* !!! Home run !!! */
+ if (likely(last_spec == nb_objs)) {
+ rte_node_next_stream_move(graph, node, next_index);
+ return nb_objs;
+ }
+
+ held += last_spec;
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ rte_node_next_stream_put(graph, node, next_index, held);
+ /* Save the last next used */
+ *(uint16_t *)node->ctx = next_index;
+
+ return nb_objs;
+}
+
+static int
+ip4_rewrite_node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+
+ RTE_SET_USED(graph);
+ RTE_SET_USED(node);
+ node_dbg("ip4_rewrite", "Initialized ip4_rewrite node initialized");
+
+ return 0;
+}
+
+static struct rte_node_register ip4_rewrite_node = {
+ .process = ip4_rewrite_node_process,
+ .name = "ip4_rewrite",
+ /* Default edge i.e '0' is pkt drop */
+ .nb_edges = 1,
+ .next_nodes = {
+ [0] = "pkt_drop",
+ },
+ .init = ip4_rewrite_node_init,
+};
+
+RTE_NODE_REGISTER(ip4_rewrite_node);
diff --git a/lib/librte_node/ip4_rewrite_priv.h b/lib/librte_node/ip4_rewrite_priv.h
new file mode 100644
index 000000000..420996a03
--- /dev/null
+++ b/lib/librte_node/ip4_rewrite_priv.h
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#ifndef __INCLUDE_IP4_REWRITE_PRIV_H__
+#define __INCLUDE_IP4_REWRITE_PRIV_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+#define RTE_GRAPH_IP4_REWRITE_MAX_NH 64
+#define RTE_GRAPH_IP4_REWRITE_MAX_LEN 56
+
+/**
+ * @internal
+ *
+ * Ipv4 rewrite next hop header data structure. Used to store port specific
+ * rewrite data.
+ */
+struct ip4_rewrite_nh_header {
+ uint16_t rewrite_len; /**< Header rewrite length. */
+ uint16_t tx_node; /**< Tx node next index identifier. */
+ uint16_t enabled; /**< NH enable flag */
+ uint16_t rsvd;
+ union {
+ struct {
+ struct rte_ether_addr dst;
+ /**< Destination mac address. */
+ struct rte_ether_addr src;
+ /**< Source mac address. */
+ };
+ uint8_t rewrite_data[RTE_GRAPH_IP4_REWRITE_MAX_LEN];
+ /**< Generic rewrite data */
+ };
+};
+
+/**
+ * @internal
+ *
+ * Ipv4 node main data structure.
+ */
+struct ip4_rewrite_node_main {
+ struct ip4_rewrite_nh_header nh[RTE_GRAPH_IP4_REWRITE_MAX_NH];
+ /**< Array of next hop header data */
+ uint16_t next_index[RTE_MAX_ETHPORTS];
+ /**< Next index of each configured port. */
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_IP4_REWRITE_PRIV_H__ */
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index ddea1237a..ef056d9af 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -2,7 +2,7 @@
# Copyright(C) 2020 Marvell International Ltd.
sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
- 'ethdev_ctrl.c')
+ 'ip4_rewrite.c', 'ethdev_ctrl.c')
headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
allow_experimental_apis = true
deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 22/26] node: add ipv4 rewrite and lookup ctrl API
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (20 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 21/26] node: add ipv4 rewrite node jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 23/26] node: add pkt drop node jerinj
` (4 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add ip4_rewrite and ip4_lookup ctrl API. ip4_lookup ctrl
API is used to add route entries for LPM lookup with
result data containing next hop id and next proto.
ip4_rewrite ctrl API is used to add rewrite data for
every next hop.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/ethdev_ctrl.c | 18 ++++++-
lib/librte_node/ip4_lookup.c | 80 ++++++++++++++++++++++++++++
lib/librte_node/ip4_rewrite.c | 56 +++++++++++++++++++
lib/librte_node/ip4_rewrite_priv.h | 22 ++++++++
lib/librte_node/rte_node_ip4_api.h | 44 +++++++++++++++
lib/librte_node/rte_node_version.map | 2 +
6 files changed, 221 insertions(+), 1 deletion(-)
diff --git a/lib/librte_node/ethdev_ctrl.c b/lib/librte_node/ethdev_ctrl.c
index 971cf0fb8..48ef8746f 100644
--- a/lib/librte_node/ethdev_ctrl.c
+++ b/lib/librte_node/ethdev_ctrl.c
@@ -10,6 +10,7 @@
#include "ethdev_rx_priv.h"
#include "ethdev_tx_priv.h"
+#include "ip4_rewrite_priv.h"
#include "node_private.h"
static struct ethdev_ctrl {
@@ -20,14 +21,17 @@ int
rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
uint16_t nb_graphs)
{
+ struct rte_node_register *ip4_rewrite_node;
struct ethdev_tx_node_main *tx_node_data;
uint16_t tx_q_used, rx_q_used, port_id;
struct rte_node_register *tx_node;
char name[RTE_NODE_NAMESIZE];
+ const char *next_nodes = name;
struct rte_mempool *mp;
+ int i, j, rc;
uint32_t id;
- int i, j;
+ ip4_rewrite_node = ip4_rewrite_node_get();
tx_node_data = ethdev_tx_node_data_get();
tx_node = ethdev_tx_node_get();
for (i = 0; i < nb_confs; i++) {
@@ -92,6 +96,18 @@ rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
node_dbg("ethdev", "Tx node %s-%s: is at %u", tx_node->name,
name, id);
+
+ /* Prepare the actual name of the cloned node */
+ snprintf(name, sizeof(name), "ethdev_tx-%u", port_id);
+
+ /* Add this tx port node as next to ip4_rewrite_node */
+ rte_node_edge_update(ip4_rewrite_node->id, RTE_EDGE_ID_INVALID,
+ &next_nodes, 1);
+ /* Assuming edge id is the last one alloc'ed */
+ rc = ip4_rewrite_set_next(
+ port_id, rte_node_edge_count(ip4_rewrite_node->id) - 1);
+ if (rc < 0)
+ return rc;
}
ctrl.nb_graphs = nb_graphs;
diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c
index c003e9c91..fb7090c40 100644
--- a/lib/librte_node/ip4_lookup.c
+++ b/lib/librte_node/ip4_lookup.c
@@ -27,6 +27,8 @@ struct ip4_lookup_node_main {
struct rte_lpm *lpm_tbl[RTE_MAX_NUMA_NODES];
};
+static struct ip4_lookup_node_main ip4_lookup_nm;
+
#if defined(RTE_MACHINE_CPUFLAG_NEON)
/* ARM64 NEON */
static uint16_t
@@ -524,12 +526,90 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
#endif
+int
+rte_node_ip4_route_add(uint32_t ip, uint8_t depth, uint16_t next_hop,
+ enum rte_node_ip4_lookup_next next_node)
+{
+ char abuf[INET6_ADDRSTRLEN];
+ struct in_addr in;
+ uint8_t socket;
+ uint32_t val;
+ int ret;
+
+ in.s_addr = htonl(ip);
+ inet_ntop(AF_INET, &in, abuf, sizeof(abuf));
+ /* Embedded next node id in next hop */
+ val = (next_node << 16) | next_hop;
+ node_dbg("ip4_lookup", "LPM: Adding route %s / %d nh (0x%x)", abuf,
+ depth, val);
+
+ for (socket = 0; socket < RTE_MAX_NUMA_NODES; socket++) {
+ if (!ip4_lookup_nm.lpm_tbl[socket])
+ continue;
+
+ ret = rte_lpm_add(ip4_lookup_nm.lpm_tbl[socket], ip, depth,
+ val);
+
+ if (ret < 0) {
+ node_err("ip4_lookup",
+ "Unable to add entry %s / %d nh (%x) to LPM table on sock %d, rc=%d\n",
+ abuf, depth, val, socket, ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int
+setup_lpm(struct ip4_lookup_node_main *nm, int socket)
+{
+ struct rte_lpm_config config_ipv4;
+ char s[64];
+
+ /* One LPM table per socket */
+ if (nm->lpm_tbl[socket])
+ return 0;
+
+ /* create the LPM table */
+ config_ipv4.max_rules = IPV4_L3FWD_LPM_MAX_RULES;
+ config_ipv4.number_tbl8s = IPV4_L3FWD_LPM_NUMBER_TBL8S;
+ config_ipv4.flags = 0;
+ snprintf(s, sizeof(s), "IPV4_L3FWD_LPM_%d", socket);
+ nm->lpm_tbl[socket] = rte_lpm_create(s, socket, &config_ipv4);
+ if (nm->lpm_tbl[socket] == NULL)
+ return -rte_errno;
+
+ return 0;
+}
+
static int
ip4_lookup_node_init(const struct rte_graph *graph, struct rte_node *node)
{
+ struct rte_lpm **lpm_p = (struct rte_lpm **)&node->ctx;
+ uint16_t socket, lcore_id;
+ static uint8_t init_once;
+ int rc;
+
RTE_SET_USED(graph);
RTE_SET_USED(node);
+ if (!init_once) {
+ /* Setup LPM tables for all sockets */
+ RTE_LCORE_FOREACH(lcore_id)
+ {
+ socket = rte_lcore_to_socket_id(lcore_id);
+ rc = setup_lpm(&ip4_lookup_nm, socket);
+ if (rc) {
+ node_err("ip4_lookup",
+ "Failed to setup lpm tbl for sock %u, rc=%d",
+ socket, rc);
+ return rc;
+ }
+ }
+ init_once = 1;
+ }
+ *lpm_p = ip4_lookup_nm.lpm_tbl[graph->socket];
node_dbg("ip4_lookup", "Initialized ip4_lookup node");
return 0;
diff --git a/lib/librte_node/ip4_rewrite.c b/lib/librte_node/ip4_rewrite.c
index f317f7f4f..14b908ec9 100644
--- a/lib/librte_node/ip4_rewrite.c
+++ b/lib/librte_node/ip4_rewrite.c
@@ -255,6 +255,56 @@ ip4_rewrite_node_init(const struct rte_graph *graph, struct rte_node *node)
return 0;
}
+int
+ip4_rewrite_set_next(uint16_t port_id, uint16_t next_index)
+{
+ if (ip4_rewrite_nm == NULL) {
+ ip4_rewrite_nm = rte_zmalloc(
+ "ip4_rewrite", sizeof(struct ip4_rewrite_node_main),
+ RTE_CACHE_LINE_SIZE);
+ if (ip4_rewrite_nm == NULL)
+ return -ENOMEM;
+ }
+ ip4_rewrite_nm->next_index[port_id] = next_index;
+
+ return 0;
+}
+
+int
+rte_node_ip4_rewrite_add(uint16_t next_hop, uint8_t *rewrite_data,
+ uint8_t rewrite_len, uint16_t dst_port)
+{
+ struct ip4_rewrite_nh_header *nh;
+
+ if (next_hop >= RTE_GRAPH_IP4_REWRITE_MAX_NH)
+ return -EINVAL;
+
+ if (rewrite_len > RTE_GRAPH_IP4_REWRITE_MAX_LEN)
+ return -EINVAL;
+
+ if (ip4_rewrite_nm == NULL) {
+ ip4_rewrite_nm = rte_zmalloc(
+ "ip4_rewrite", sizeof(struct ip4_rewrite_node_main),
+ RTE_CACHE_LINE_SIZE);
+ if (ip4_rewrite_nm == NULL)
+ return -ENOMEM;
+ }
+
+ /* Check if dst port doesn't exist as edge */
+ if (!ip4_rewrite_nm->next_index[dst_port])
+ return -EINVAL;
+
+ /* Update next hop */
+ nh = &ip4_rewrite_nm->nh[next_hop];
+
+ memcpy(nh->rewrite_data, rewrite_data, rewrite_len);
+ nh->tx_node = ip4_rewrite_nm->next_index[dst_port];
+ nh->rewrite_len = rewrite_len;
+ nh->enabled = true;
+
+ return 0;
+}
+
static struct rte_node_register ip4_rewrite_node = {
.process = ip4_rewrite_node_process,
.name = "ip4_rewrite",
@@ -266,4 +316,10 @@ static struct rte_node_register ip4_rewrite_node = {
.init = ip4_rewrite_node_init,
};
+struct rte_node_register *
+ip4_rewrite_node_get(void)
+{
+ return &ip4_rewrite_node;
+}
+
RTE_NODE_REGISTER(ip4_rewrite_node);
diff --git a/lib/librte_node/ip4_rewrite_priv.h b/lib/librte_node/ip4_rewrite_priv.h
index 420996a03..80f0abdc9 100644
--- a/lib/librte_node/ip4_rewrite_priv.h
+++ b/lib/librte_node/ip4_rewrite_priv.h
@@ -48,6 +48,28 @@ struct ip4_rewrite_node_main {
/**< Next index of each configured port. */
};
+/**
+ * @internal
+ *
+ * Get the ipv4 rewrite node.
+ *
+ * @retrun
+ * Pointer to the ipv4 rewrite node.
+ */
+struct rte_node_register *ip4_rewrite_node_get(void);
+
+/**
+ * @internal
+ *
+ * Set the Edge index of a given port_id.
+ *
+ * @param port_id
+ * Ethernet port identifier.
+ * @param next_index
+ * Edge index of the Given Tx node.
+ */
+int ip4_rewrite_set_next(uint16_t port_id, uint16_t next_index);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/librte_node/rte_node_ip4_api.h b/lib/librte_node/rte_node_ip4_api.h
index 37c12bf82..394cac097 100644
--- a/lib/librte_node/rte_node_ip4_api.h
+++ b/lib/librte_node/rte_node_ip4_api.h
@@ -36,6 +36,50 @@ enum rte_node_ip4_lookup_next {
/**< Number of next nodes of lookup node. */
};
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add ipv4 route to lookup table.
+ *
+ * @param ip
+ * IP address of route to be added.
+ * @param depth
+ * Depth of the rule to be added.
+ * @param next_hop
+ * Next hop id of the rule result to be added.
+ * @param next_node
+ * Next node to redirect traffic to.
+ *
+ * @return
+ * 0 on success, negative otherwise.
+ */
+__rte_experimental
+int rte_node_ip4_route_add(uint32_t ip, uint8_t depth, uint16_t next_hop,
+ enum rte_node_ip4_lookup_next next_node);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add a next hop's rewrite data.
+ *
+ * @param next_hop
+ * Next hop id to add rewrite data to.
+ * @param rewrite_data
+ * Rewrite data.
+ * @param rewrite_len
+ * Length of rewrite data.
+ * @param dst_port
+ * Destination port to redirect traffic to.
+ *
+ * @return
+ * 0 on success, negative otherwise.
+ */
+__rte_experimental
+int rte_node_ip4_rewrite_add(uint16_t next_hop, uint8_t *rewrite_data,
+ uint8_t rewrite_len, uint16_t dst_port);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/librte_node/rte_node_version.map b/lib/librte_node/rte_node_version.map
index c6c71bd02..a799b0d38 100644
--- a/lib/librte_node/rte_node_version.map
+++ b/lib/librte_node/rte_node_version.map
@@ -2,6 +2,8 @@ EXPERIMENTAL {
global:
rte_node_eth_config;
+ rte_node_ip4_route_add;
+ rte_node_ip4_rewrite_add;
rte_node_logtype;
local: *;
};
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 23/26] node: add pkt drop node
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (21 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 22/26] node: add ipv4 rewrite and lookup ctrl API jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 24/26] l3fwd-graph: add graph based l3fwd skeleton jerinj
` (3 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add pkt drop node process function for pkt_drop
rte_node. This node simply free's every object received as
an rte_mbuf to its rte_pktmbuf pool.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/Makefile | 1 +
lib/librte_node/meson.build | 2 +-
lib/librte_node/pkt_drop.c | 26 ++++++++++++++++++++++++++
3 files changed, 28 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_node/pkt_drop.c
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index be7e7e338..aaf041580 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -21,6 +21,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_rewrite.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += pkt_drop.c
# install header files
SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index ef056d9af..59e11e5b4 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -2,7 +2,7 @@
# Copyright(C) 2020 Marvell International Ltd.
sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
- 'ip4_rewrite.c', 'ethdev_ctrl.c')
+ 'ip4_rewrite.c', 'pkt_drop.c', 'ethdev_ctrl.c')
headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
allow_experimental_apis = true
deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
diff --git a/lib/librte_node/pkt_drop.c b/lib/librte_node/pkt_drop.c
new file mode 100644
index 000000000..c35001323
--- /dev/null
+++ b/lib/librte_node/pkt_drop.c
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_graph.h>
+#include <rte_mbuf.h>
+
+static uint16_t
+pkt_drop_process(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ RTE_SET_USED(node);
+ RTE_SET_USED(graph);
+
+ rte_pktmbuf_free_bulk((struct rte_mbuf **)objs, nb_objs);
+
+ return nb_objs;
+}
+
+static struct rte_node_register pkt_drop_node = {
+ .process = pkt_drop_process,
+ .name = "pkt_drop",
+};
+
+RTE_NODE_REGISTER(pkt_drop_node);
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 24/26] l3fwd-graph: add graph based l3fwd skeleton
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (22 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 23/26] node: add pkt drop node jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 25/26] l3fwd-graph: add ethdev configuration changes jerinj
` (2 subsequent siblings)
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Thomas Monjalon, Marko Kovacevic, Ori Kam, Bruce Richardson,
Radu Nicolau, Akhil Goyal, Tomasz Kantecki, Sunil Kumar Kori,
Pavan Nikhilesh, Nithin Dabilpuram
Cc: dev, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add graph based l3fwd application skeleton with cmdline
parsing support inline with normal l3fwd.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
MAINTAINERS | 3 +
examples/Makefile | 3 +
examples/l3fwd-graph/Makefile | 58 ++++
examples/l3fwd-graph/main.c | 528 +++++++++++++++++++++++++++++++
examples/l3fwd-graph/meson.build | 13 +
examples/meson.build | 6 +-
6 files changed, 609 insertions(+), 2 deletions(-)
create mode 100644 examples/l3fwd-graph/Makefile
create mode 100644 examples/l3fwd-graph/main.c
create mode 100644 examples/l3fwd-graph/meson.build
diff --git a/MAINTAINERS b/MAINTAINERS
index 3959ed19a..73debcd6d 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1580,6 +1580,9 @@ F: doc/guides/sample_app_ug/l2_forward_event.rst
F: examples/l3fwd/
F: doc/guides/sample_app_ug/l3_forward.rst
+M: Nithin Dabilpuram <ndabilpuram@marvell.com>
+F: examples/l3fwd-graph/
+
F: examples/link_status_interrupt/
F: doc/guides/sample_app_ug/link_status_intr.rst
diff --git a/examples/Makefile b/examples/Makefile
index feff79784..b7e99a2f7 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -51,6 +51,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_ACL) += l3fwd-acl
ifeq ($(CONFIG_RTE_LIBRTE_LPM)$(CONFIG_RTE_LIBRTE_HASH),yy)
DIRS-$(CONFIG_RTE_LIBRTE_POWER) += l3fwd-power
endif
+ifeq ($(CONFIG_RTE_LIBRTE_GRAPH),y)
+DIRS-y += l3fwd-graph
+endif
DIRS-y += link_status_interrupt
DIRS-y += multi_process
DIRS-y += ntb
diff --git a/examples/l3fwd-graph/Makefile b/examples/l3fwd-graph/Makefile
new file mode 100644
index 000000000..f3cf275ec
--- /dev/null
+++ b/examples/l3fwd-graph/Makefile
@@ -0,0 +1,58 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+# binary name
+APP = l3fwd-graph
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+# Build using pkg-config variables if possible
+ifeq ($(shell pkg-config --exists libdpdk && echo 0),0)
+
+all: shared
+.PHONY: shared static
+shared: build/$(APP)-shared
+ ln -sf $(APP)-shared build/$(APP)
+static: build/$(APP)-static
+ ln -sf $(APP)-static build/$(APP)
+
+PKGCONF=pkg-config --define-prefix
+
+PC_FILE := $(shell $(PKGCONF) --path libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -DALLOW_EXPERIMENTAL_API
+LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
+LDFLAGS_STATIC = -Wl,-Bstatic $(shell $(PKGCONF) --static --libs libdpdk)
+
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+ $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
+
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+ $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
+
+build:
+ @mkdir -p $@
+
+.PHONY: clean
+clean:
+ rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+ test -d build && rmdir -p build || true
+
+else # Build using legacy build system
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, detect a build directory, by looking for a path with a .config
+RTE_TARGET ?= $(notdir $(abspath $(dir $(firstword $(wildcard $(RTE_SDK)/*/.config)))))
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(SRCDIR)
+CFLAGS += -O3 $(USER_FLAGS)
+CFLAGS += $(WERROR_FLAGS)
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+endif
diff --git a/examples/l3fwd-graph/main.c b/examples/l3fwd-graph/main.c
new file mode 100644
index 000000000..8cee8bebd
--- /dev/null
+++ b/examples/l3fwd-graph/main.c
@@ -0,0 +1,528 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <arpa/inet.h>
+#include <errno.h>
+#include <getopt.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <unistd.h>
+#include <sys/queue.h>
+#include <sys/types.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_eal.h>
+#include <rte_ethdev.h>
+#include <rte_log.h>
+#include <rte_mempool.h>
+#include <rte_per_lcore.h>
+#include <rte_string_fns.h>
+#include <rte_vect.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_parse_etheraddr.h>
+
+/* Log type */
+#define RTE_LOGTYPE_L3FWD_GRAPH RTE_LOGTYPE_USER1
+
+/*
+ * Configurable number of RX/TX ring descriptors
+ */
+#define RTE_TEST_RX_DESC_DEFAULT 1024
+#define RTE_TEST_TX_DESC_DEFAULT 1024
+
+#define MAX_TX_QUEUE_PER_PORT RTE_MAX_ETHPORTS
+#define MAX_RX_QUEUE_PER_PORT 128
+
+#define MAX_RX_QUEUE_PER_LCORE 16
+
+#define MAX_LCORE_PARAMS 1024
+
+#define NB_SOCKETS 8
+
+/**< Ports set in promiscuous mode off by default. */
+static int promiscuous_on;
+
+static int numa_on = 1; /**< NUMA is enabled by default. */
+static int per_port_pool; /**< Use separate buffer pools per port; disabled */
+ /**< by default */
+
+static volatile bool force_quit;
+
+/* Ethernet addresses of ports */
+static uint64_t dest_eth_addr[RTE_MAX_ETHPORTS];
+xmm_t val_eth[RTE_MAX_ETHPORTS];
+
+/* Mask of enabled ports */
+static uint32_t enabled_port_mask;
+
+struct lcore_rx_queue {
+ uint16_t port_id;
+ uint8_t queue_id;
+};
+
+/* Lcore conf */
+struct lcore_conf {
+ uint16_t n_rx_queue;
+ struct lcore_rx_queue rx_queue_list[MAX_RX_QUEUE_PER_LCORE];
+} __rte_cache_aligned;
+
+static struct lcore_conf lcore_conf[RTE_MAX_LCORE];
+
+struct lcore_params {
+ uint16_t port_id;
+ uint8_t queue_id;
+ uint8_t lcore_id;
+} __rte_cache_aligned;
+
+static struct lcore_params lcore_params_array[MAX_LCORE_PARAMS];
+static struct lcore_params lcore_params_array_default[] = {
+ {0, 0, 2}, {0, 1, 2}, {0, 2, 2}, {1, 0, 2}, {1, 1, 2},
+ {1, 2, 2}, {2, 0, 2}, {3, 0, 3}, {3, 1, 3},
+};
+
+static struct lcore_params *lcore_params = lcore_params_array_default;
+static uint16_t nb_lcore_params = RTE_DIM(lcore_params_array_default);
+
+static struct rte_eth_conf port_conf = {
+ .rxmode = {
+ .mq_mode = ETH_MQ_RX_RSS,
+ .max_rx_pkt_len = RTE_ETHER_MAX_LEN,
+ .split_hdr_size = 0,
+ },
+ .rx_adv_conf = {
+ .rss_conf = {
+ .rss_key = NULL,
+ .rss_hf = ETH_RSS_IP,
+ },
+ },
+ .txmode = {
+ .mq_mode = ETH_MQ_TX_NONE,
+ },
+};
+
+static int
+check_lcore_params(void)
+{
+ uint8_t queue, lcore;
+ int socketid;
+ uint16_t i;
+
+ for (i = 0; i < nb_lcore_params; ++i) {
+ queue = lcore_params[i].queue_id;
+ if (queue >= MAX_RX_QUEUE_PER_PORT) {
+ printf("Invalid queue number: %hhu\n", queue);
+ return -1;
+ }
+ lcore = lcore_params[i].lcore_id;
+ if (!rte_lcore_is_enabled(lcore)) {
+ printf("Error: lcore %hhu is not enabled in lcore mask\n",
+ lcore);
+ return -1;
+ }
+
+ if (lcore == rte_get_master_lcore()) {
+ printf("Error: lcore %u is master lcore\n", lcore);
+ return -1;
+ }
+ socketid = rte_lcore_to_socket_id(lcore);
+ if ((socketid != 0) && (numa_on == 0)) {
+ printf("Warning: lcore %hhu is on socket %d with numa off\n",
+ lcore, socketid);
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_port_config(void)
+{
+ uint16_t portid;
+ uint16_t i;
+
+ for (i = 0; i < nb_lcore_params; ++i) {
+ portid = lcore_params[i].port_id;
+ if ((enabled_port_mask & (1 << portid)) == 0) {
+ printf("Port %u is not enabled in port mask\n", portid);
+ return -1;
+ }
+ if (!rte_eth_dev_is_valid_port(portid)) {
+ printf("Port %u is not present on the board\n", portid);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+init_lcore_rx_queues(void)
+{
+ uint16_t i, nb_rx_queue;
+ uint8_t lcore;
+
+ for (i = 0; i < nb_lcore_params; ++i) {
+ lcore = lcore_params[i].lcore_id;
+ nb_rx_queue = lcore_conf[lcore].n_rx_queue;
+ if (nb_rx_queue >= MAX_RX_QUEUE_PER_LCORE) {
+ printf("Error: too many queues (%u) for lcore: %u\n",
+ (unsigned int)nb_rx_queue + 1,
+ (unsigned int)lcore);
+ return -1;
+ }
+
+ lcore_conf[lcore].rx_queue_list[nb_rx_queue].port_id =
+ lcore_params[i].port_id;
+ lcore_conf[lcore].rx_queue_list[nb_rx_queue].queue_id =
+ lcore_params[i].queue_id;
+ lcore_conf[lcore].n_rx_queue++;
+ }
+
+ return 0;
+}
+
+/* Display usage */
+static void
+print_usage(const char *prgname)
+{
+ fprintf(stderr,
+ "%s [EAL options] --"
+ " -p PORTMASK"
+ " [-P]"
+ " [-E]"
+ " [-L]"
+ " --config (port,queue,lcore)[,(port,queue,lcore)]"
+ " [--eth-dest=X,MM:MM:MM:MM:MM:MM]"
+ " [--enable-jumbo [--max-pkt-len PKTLEN]]"
+ " [--no-numa]"
+ " [--per-port-pool]\n\n"
+
+ " -p PORTMASK: Hexadecimal bitmask of ports to configure\n"
+ " -P : Enable promiscuous mode\n"
+ " --config (port,queue,lcore): Rx queue configuration\n"
+ " --eth-dest=X,MM:MM:MM:MM:MM:MM: Ethernet destination for "
+ "port X\n"
+ " --enable-jumbo: Enable jumbo frames\n"
+ " --max-pkt-len: Under the premise of enabling jumbo,\n"
+ " maximum packet length in decimal (64-9600)\n"
+ " --no-numa: Disable numa awareness\n"
+ " --per-port-pool: Use separate buffer pool per port\n\n",
+ prgname);
+}
+
+static int
+parse_max_pkt_len(const char *pktlen)
+{
+ unsigned long len;
+ char *end = NULL;
+
+ /* Parse decimal string */
+ len = strtoul(pktlen, &end, 10);
+ if ((pktlen[0] == '\0') || (end == NULL) || (*end != '\0'))
+ return -1;
+
+ if (len == 0)
+ return -1;
+
+ return len;
+}
+
+static int
+parse_portmask(const char *portmask)
+{
+ char *end = NULL;
+ unsigned long pm;
+
+ /* Parse hexadecimal string */
+ pm = strtoul(portmask, &end, 16);
+ if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
+ return -1;
+
+ if (pm == 0)
+ return -1;
+
+ return pm;
+}
+
+static int
+parse_config(const char *q_arg)
+{
+ enum fieldnames { FLD_PORT = 0, FLD_QUEUE, FLD_LCORE, _NUM_FLD };
+ unsigned long int_fld[_NUM_FLD];
+ const char *p, *p0 = q_arg;
+ char *str_fld[_NUM_FLD];
+ uint32_t size;
+ char s[256];
+ char *end;
+ int i;
+
+ nb_lcore_params = 0;
+
+ while ((p = strchr(p0, '(')) != NULL) {
+ ++p;
+ p0 = strchr(p, ')');
+ if (p0 == NULL)
+ return -1;
+
+ size = p0 - p;
+ if (size >= sizeof(s))
+ return -1;
+
+ snprintf(s, sizeof(s), "%.*s", size, p);
+ if (rte_strsplit(s, sizeof(s), str_fld, _NUM_FLD, ',') !=
+ _NUM_FLD)
+ return -1;
+ for (i = 0; i < _NUM_FLD; i++) {
+ errno = 0;
+ int_fld[i] = strtoul(str_fld[i], &end, 0);
+ if (errno != 0 || end == str_fld[i] || int_fld[i] > 255)
+ return -1;
+ }
+ if (nb_lcore_params >= MAX_LCORE_PARAMS) {
+ printf("Exceeded max number of lcore params: %hu\n",
+ nb_lcore_params);
+ return -1;
+ }
+ lcore_params_array[nb_lcore_params].port_id =
+ (uint8_t)int_fld[FLD_PORT];
+ lcore_params_array[nb_lcore_params].queue_id =
+ (uint8_t)int_fld[FLD_QUEUE];
+ lcore_params_array[nb_lcore_params].lcore_id =
+ (uint8_t)int_fld[FLD_LCORE];
+ ++nb_lcore_params;
+ }
+ lcore_params = lcore_params_array;
+
+ return 0;
+}
+
+static void
+parse_eth_dest(const char *optarg)
+{
+ uint8_t c, *dest, peer_addr[6];
+ uint16_t portid;
+ char *port_end;
+
+ errno = 0;
+ portid = strtoul(optarg, &port_end, 10);
+ if (errno != 0 || port_end == optarg || *port_end++ != ',')
+ rte_exit(EXIT_FAILURE, "Invalid eth-dest: %s", optarg);
+ if (portid >= RTE_MAX_ETHPORTS)
+ rte_exit(EXIT_FAILURE,
+ "eth-dest: port %d >= RTE_MAX_ETHPORTS(%d)\n", portid,
+ RTE_MAX_ETHPORTS);
+
+ if (cmdline_parse_etheraddr(NULL, port_end, &peer_addr,
+ sizeof(peer_addr)) < 0)
+ rte_exit(EXIT_FAILURE, "Invalid ethernet address: %s\n",
+ port_end);
+ dest = (uint8_t *)&dest_eth_addr[portid];
+ for (c = 0; c < 6; c++)
+ dest[c] = peer_addr[c];
+ *(uint64_t *)(val_eth + portid) = dest_eth_addr[portid];
+}
+
+#define MAX_JUMBO_PKT_LEN 9600
+#define MEMPOOL_CACHE_SIZE 256
+
+static const char short_options[] = "p:" /* portmask */
+ "P" /* promiscuous */
+ "L" /* enable long prefix match */
+ "E" /* enable exact match */
+ ;
+
+#define CMD_LINE_OPT_CONFIG "config"
+#define CMD_LINE_OPT_ETH_DEST "eth-dest"
+#define CMD_LINE_OPT_NO_NUMA "no-numa"
+#define CMD_LINE_OPT_ENABLE_JUMBO "enable-jumbo"
+#define CMD_LINE_OPT_PER_PORT_POOL "per-port-pool"
+enum {
+ /* Long options mapped to a short option */
+
+ /* First long only option value must be >= 256, so that we won't
+ * conflict with short options
+ */
+ CMD_LINE_OPT_MIN_NUM = 256,
+ CMD_LINE_OPT_CONFIG_NUM,
+ CMD_LINE_OPT_ETH_DEST_NUM,
+ CMD_LINE_OPT_NO_NUMA_NUM,
+ CMD_LINE_OPT_ENABLE_JUMBO_NUM,
+ CMD_LINE_OPT_PARSE_PER_PORT_POOL,
+};
+
+static const struct option lgopts[] = {
+ {CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
+ {CMD_LINE_OPT_ETH_DEST, 1, 0, CMD_LINE_OPT_ETH_DEST_NUM},
+ {CMD_LINE_OPT_NO_NUMA, 0, 0, CMD_LINE_OPT_NO_NUMA_NUM},
+ {CMD_LINE_OPT_ENABLE_JUMBO, 0, 0, CMD_LINE_OPT_ENABLE_JUMBO_NUM},
+ {CMD_LINE_OPT_PER_PORT_POOL, 0, 0, CMD_LINE_OPT_PARSE_PER_PORT_POOL},
+ {NULL, 0, 0, 0},
+};
+
+/*
+ * This expression is used to calculate the number of mbufs needed
+ * depending on user input, taking into account memory for rx and
+ * tx hardware rings, cache per lcore and mtable per port per lcore.
+ * RTE_MAX is used to ensure that NB_MBUF never goes below a minimum
+ * value of 8192
+ */
+#define NB_MBUF(nports) \
+ RTE_MAX((nports * nb_rx_queue * nb_rxd + \
+ nports * nb_lcores * RTE_GRAPH_BURST_SIZE + \
+ nports * n_tx_queue * nb_txd + \
+ nb_lcores * MEMPOOL_CACHE_SIZE), 8192u)
+
+/* Parse the argument given in the command line of the application */
+static int
+parse_args(int argc, char **argv)
+{
+ char *prgname = argv[0];
+ int option_index;
+ char **argvopt;
+ int opt, ret;
+
+ argvopt = argv;
+
+ /* Error or normal output strings. */
+ while ((opt = getopt_long(argc, argvopt, short_options, lgopts,
+ &option_index)) != EOF) {
+
+ switch (opt) {
+ /* Portmask */
+ case 'p':
+ enabled_port_mask = parse_portmask(optarg);
+ if (enabled_port_mask == 0) {
+ fprintf(stderr, "Invalid portmask\n");
+ print_usage(prgname);
+ return -1;
+ }
+ break;
+
+ case 'P':
+ promiscuous_on = 1;
+ break;
+
+ /* Long options */
+ case CMD_LINE_OPT_CONFIG_NUM:
+ ret = parse_config(optarg);
+ if (ret) {
+ fprintf(stderr, "Invalid config\n");
+ print_usage(prgname);
+ return -1;
+ }
+ break;
+
+ case CMD_LINE_OPT_ETH_DEST_NUM:
+ parse_eth_dest(optarg);
+ break;
+
+ case CMD_LINE_OPT_NO_NUMA_NUM:
+ numa_on = 0;
+ break;
+
+ case CMD_LINE_OPT_ENABLE_JUMBO_NUM: {
+ const struct option lenopts = {"max-pkt-len",
+ required_argument, 0, 0};
+
+ port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
+ port_conf.txmode.offloads |= DEV_TX_OFFLOAD_MULTI_SEGS;
+
+ /*
+ * if no max-pkt-len set, use the default
+ * value RTE_ETHER_MAX_LEN.
+ */
+ if (getopt_long(argc, argvopt, "", &lenopts,
+ &option_index) == 0) {
+ ret = parse_max_pkt_len(optarg);
+ if (ret < 64 || ret > MAX_JUMBO_PKT_LEN) {
+ fprintf(stderr, "Invalid maximum "
+ "packet length\n");
+ print_usage(prgname);
+ return -1;
+ }
+ port_conf.rxmode.max_rx_pkt_len = ret;
+ }
+ break;
+ }
+
+ case CMD_LINE_OPT_PARSE_PER_PORT_POOL:
+ printf("Per port buffer pool is enabled\n");
+ per_port_pool = 1;
+ break;
+
+ default:
+ print_usage(prgname);
+ return -1;
+ }
+ }
+
+ if (optind >= 0)
+ argv[optind - 1] = prgname;
+ ret = optind - 1;
+ optind = 1; /* Reset getopt lib */
+
+ return ret;
+}
+
+static void
+signal_handler(int signum)
+{
+ if (signum == SIGINT || signum == SIGTERM) {
+ printf("\n\nSignal %d received, preparing to exit...\n",
+ signum);
+ force_quit = true;
+ }
+}
+
+int
+main(int argc, char **argv)
+{
+ uint16_t portid;
+ int ret;
+
+ /* Init EAL */
+ ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Invalid EAL parameters\n");
+ argc -= ret;
+ argv += ret;
+
+ force_quit = false;
+ signal(SIGINT, signal_handler);
+ signal(SIGTERM, signal_handler);
+
+ /* Pre-init dst MACs for all ports to 02:00:00:00:00:xx */
+ for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) {
+ dest_eth_addr[portid] =
+ RTE_ETHER_LOCAL_ADMIN_ADDR + ((uint64_t)portid << 40);
+ *(uint64_t *)(val_eth + portid) = dest_eth_addr[portid];
+ }
+
+ /* Parse application arguments (after the EAL ones) */
+ ret = parse_args(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Invalid L3FWD_GRAPH parameters\n");
+
+ if (check_lcore_params() < 0)
+ rte_exit(EXIT_FAILURE, "check_lcore_params() failed\n");
+
+ ret = init_lcore_rx_queues();
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "init_lcore_rx_queues() failed\n");
+
+ if (check_port_config() < 0)
+ rte_exit(EXIT_FAILURE, "check_port_config() failed\n");
+
+ printf("Bye...\n");
+
+ return ret;
+}
diff --git a/examples/l3fwd-graph/meson.build b/examples/l3fwd-graph/meson.build
new file mode 100644
index 000000000..a816bd890
--- /dev/null
+++ b/examples/l3fwd-graph/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+deps += ['graph', 'eal', 'lpm', 'ethdev', 'node' ]
+sources = files(
+ 'main.c'
+)
+allow_experimental_apis = true
diff --git a/examples/meson.build b/examples/meson.build
index 1f2b6f516..3b540012f 100644
--- a/examples/meson.build
+++ b/examples/meson.build
@@ -2,8 +2,10 @@
# Copyright(c) 2017-2019 Intel Corporation
driver_libs = []
+node_libs = []
if get_option('default_library') == 'static'
driver_libs = dpdk_drivers
+ node_libs = dpdk_graph_nodes
endif
execinfo = cc.find_library('execinfo', required: false)
@@ -23,7 +25,7 @@ all_examples = [
'l2fwd', 'l2fwd-cat', 'l2fwd-event',
'l2fwd-crypto', 'l2fwd-jobstats',
'l2fwd-keepalive', 'l3fwd',
- 'l3fwd-acl', 'l3fwd-power',
+ 'l3fwd-acl', 'l3fwd-power', 'l3fwd-graph',
'link_status_interrupt',
'multi_process/client_server_mp/mp_client',
'multi_process/client_server_mp/mp_server',
@@ -99,7 +101,7 @@ foreach example: examples
endif
executable('dpdk-' + name, sources,
include_directories: includes,
- link_whole: driver_libs,
+ link_whole: driver_libs + node_libs,
link_args: dpdk_extra_ldflags,
c_args: cflags,
dependencies: dep_objs)
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 25/26] l3fwd-graph: add ethdev configuration changes
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (23 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 24/26] l3fwd-graph: add graph based l3fwd skeleton jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 26/26] l3fwd-graph: add graph config and main loop jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Marko Kovacevic, Ori Kam, Bruce Richardson, Radu Nicolau,
Akhil Goyal, Tomasz Kantecki, Sunil Kumar Kori, Pavan Nikhilesh,
Nithin Dabilpuram
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add changes to ethdev port and queue configuration based
on command line parameters for l3fwd graph application.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
examples/l3fwd-graph/main.c | 350 +++++++++++++++++++++++++++++++++++-
1 file changed, 349 insertions(+), 1 deletion(-)
diff --git a/examples/l3fwd-graph/main.c b/examples/l3fwd-graph/main.c
index 8cee8bebd..436987877 100644
--- a/examples/l3fwd-graph/main.c
+++ b/examples/l3fwd-graph/main.c
@@ -19,8 +19,10 @@
#include <rte_branch_prediction.h>
#include <rte_common.h>
+#include <rte_cycles.h>
#include <rte_eal.h>
#include <rte_ethdev.h>
+#include <rte_lcore.h>
#include <rte_log.h>
#include <rte_mempool.h>
#include <rte_per_lcore.h>
@@ -48,6 +50,10 @@
#define NB_SOCKETS 8
+/* Static global variables used within this file. */
+static uint16_t nb_rxd = RTE_TEST_RX_DESC_DEFAULT;
+static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT;
+
/**< Ports set in promiscuous mode off by default. */
static int promiscuous_on;
@@ -59,6 +65,7 @@ static volatile bool force_quit;
/* Ethernet addresses of ports */
static uint64_t dest_eth_addr[RTE_MAX_ETHPORTS];
+static struct rte_ether_addr ports_eth_addr[RTE_MAX_ETHPORTS];
xmm_t val_eth[RTE_MAX_ETHPORTS];
/* Mask of enabled ports */
@@ -109,6 +116,8 @@ static struct rte_eth_conf port_conf = {
},
};
+static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
+
static int
check_lcore_params(void)
{
@@ -164,6 +173,27 @@ check_port_config(void)
return 0;
}
+static uint8_t
+get_port_n_rx_queues(const uint16_t port)
+{
+ int queue = -1;
+ uint16_t i;
+
+ for (i = 0; i < nb_lcore_params; ++i) {
+ if (lcore_params[i].port_id == port) {
+ if (lcore_params[i].queue_id == queue + 1)
+ queue = lcore_params[i].queue_id;
+ else
+ rte_exit(EXIT_FAILURE,
+ "Queue ids of the port %d must be"
+ " in sequence and must start with 0\n",
+ lcore_params[i].port_id);
+ }
+ }
+
+ return (uint8_t)(++queue);
+}
+
static int
init_lcore_rx_queues(void)
{
@@ -473,6 +503,120 @@ parse_args(int argc, char **argv)
return ret;
}
+static void
+print_ethaddr(const char *name, const struct rte_ether_addr *eth_addr)
+{
+ char buf[RTE_ETHER_ADDR_FMT_SIZE];
+ rte_ether_format_addr(buf, RTE_ETHER_ADDR_FMT_SIZE, eth_addr);
+ printf("%s%s", name, buf);
+}
+
+static int
+init_mem(uint16_t portid, uint32_t nb_mbuf)
+{
+ uint32_t lcore_id;
+ int socketid;
+ char s[64];
+
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+
+ if (numa_on)
+ socketid = rte_lcore_to_socket_id(lcore_id);
+ else
+ socketid = 0;
+
+ if (socketid >= NB_SOCKETS) {
+ rte_exit(EXIT_FAILURE,
+ "Socket %d of lcore %u is out of range %d\n",
+ socketid, lcore_id, NB_SOCKETS);
+ }
+
+ if (pktmbuf_pool[portid][socketid] == NULL) {
+ snprintf(s, sizeof(s), "mbuf_pool_%d:%d", portid,
+ socketid);
+ /* Create a pool with priv size of a cacheline */
+ pktmbuf_pool[portid][socketid] =
+ rte_pktmbuf_pool_create(
+ s, nb_mbuf, MEMPOOL_CACHE_SIZE,
+ RTE_CACHE_LINE_SIZE,
+ RTE_MBUF_DEFAULT_BUF_SIZE, socketid);
+ if (pktmbuf_pool[portid][socketid] == NULL)
+ rte_exit(EXIT_FAILURE,
+ "Cannot init mbuf pool on socket %d\n",
+ socketid);
+ else
+ printf("Allocated mbuf pool on socket %d\n",
+ socketid);
+ }
+ }
+
+ return 0;
+}
+
+/* Check the link status of all ports in up to 9s, and print them finally */
+static void
+check_all_ports_link_status(uint32_t port_mask)
+{
+#define CHECK_INTERVAL 100 /* 100ms */
+#define MAX_CHECK_TIME 90 /* 9s (90 * 100ms) in total */
+ uint8_t count, all_ports_up, print_flag = 0;
+ struct rte_eth_link link;
+ uint16_t portid;
+
+ printf("\nChecking link status");
+ fflush(stdout);
+ for (count = 0; count <= MAX_CHECK_TIME; count++) {
+ if (force_quit)
+ return;
+ all_ports_up = 1;
+ RTE_ETH_FOREACH_DEV(portid)
+ {
+ if (force_quit)
+ return;
+ if ((port_mask & (1 << portid)) == 0)
+ continue;
+ memset(&link, 0, sizeof(link));
+ rte_eth_link_get_nowait(portid, &link);
+ /* Print link status if flag set */
+ if (print_flag == 1) {
+ if (link.link_status)
+ printf("Port%d Link Up. Speed %u Mbps "
+ "-%s\n",
+ portid, link.link_speed,
+ (link.link_duplex ==
+ ETH_LINK_FULL_DUPLEX)
+ ? ("full-duplex")
+ : ("half-duplex\n"));
+ else
+ printf("Port %d Link Down\n", portid);
+ continue;
+ }
+ /* Clear all_ports_up flag if any link down */
+ if (link.link_status == ETH_LINK_DOWN) {
+ all_ports_up = 0;
+ break;
+ }
+ }
+ /* After finally printing all link status, get out */
+ if (print_flag == 1)
+ break;
+
+ if (all_ports_up == 0) {
+ printf(".");
+ fflush(stdout);
+ rte_delay_ms(CHECK_INTERVAL);
+ }
+
+ /* Set the print_flag if all ports up or timeout */
+ if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) {
+ print_flag = 1;
+ printf("Done\n");
+ }
+ }
+}
+
static void
signal_handler(int signum)
{
@@ -486,7 +630,14 @@ signal_handler(int signum)
int
main(int argc, char **argv)
{
- uint16_t portid;
+ uint8_t nb_rx_queue, queue, socketid;
+ struct rte_eth_dev_info dev_info;
+ uint32_t n_tx_queue, nb_lcores;
+ struct rte_eth_txconf *txconf;
+ uint16_t queueid, portid;
+ struct lcore_conf *qconf;
+ uint32_t lcore_id;
+ uint32_t nb_ports;
int ret;
/* Init EAL */
@@ -522,6 +673,203 @@ main(int argc, char **argv)
if (check_port_config() < 0)
rte_exit(EXIT_FAILURE, "check_port_config() failed\n");
+ nb_ports = rte_eth_dev_count_avail();
+ nb_lcores = rte_lcore_count();
+
+ /* Initialize all ports */
+ RTE_ETH_FOREACH_DEV(portid)
+ {
+ struct rte_eth_conf local_port_conf = port_conf;
+
+ /* Skip ports that are not enabled */
+ if ((enabled_port_mask & (1 << portid)) == 0) {
+ printf("\nSkipping disabled port %d\n", portid);
+ continue;
+ }
+
+ /* Init port */
+ printf("Initializing port %d ... ", portid);
+ fflush(stdout);
+
+ nb_rx_queue = get_port_n_rx_queues(portid);
+ n_tx_queue = nb_lcores;
+ if (n_tx_queue > MAX_TX_QUEUE_PER_PORT)
+ n_tx_queue = MAX_TX_QUEUE_PER_PORT;
+ printf("Creating queues: nb_rxq=%d nb_txq=%u... ",
+ nb_rx_queue, n_tx_queue);
+
+ rte_eth_dev_info_get(portid, &dev_info);
+ if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
+ local_port_conf.txmode.offloads |=
+ DEV_TX_OFFLOAD_MBUF_FAST_FREE;
+
+ local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
+ dev_info.flow_type_rss_offloads;
+ if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
+ port_conf.rx_adv_conf.rss_conf.rss_hf) {
+ printf("Port %u modified RSS hash function based on "
+ "hardware support,"
+ "requested:%#" PRIx64 " configured:%#" PRIx64
+ "\n",
+ portid, port_conf.rx_adv_conf.rss_conf.rss_hf,
+ local_port_conf.rx_adv_conf.rss_conf.rss_hf);
+ }
+
+ ret = rte_eth_dev_configure(portid, nb_rx_queue,
+ n_tx_queue, &local_port_conf);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "Cannot configure device: err=%d, port=%d\n",
+ ret, portid);
+
+ ret = rte_eth_dev_adjust_nb_rx_tx_desc(portid, &nb_rxd,
+ &nb_txd);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "Cannot adjust number of descriptors: err=%d, "
+ "port=%d\n",
+ ret, portid);
+
+ rte_eth_macaddr_get(portid, &ports_eth_addr[portid]);
+ print_ethaddr(" Address:", &ports_eth_addr[portid]);
+ printf(", ");
+ print_ethaddr(
+ "Destination:",
+ (const struct rte_ether_addr *)&dest_eth_addr[portid]);
+ printf(", ");
+
+ /*
+ * prepare src MACs for each port.
+ */
+ rte_ether_addr_copy(
+ &ports_eth_addr[portid],
+ (struct rte_ether_addr *)(val_eth + portid) + 1);
+
+ /* Init memory */
+ if (!per_port_pool) {
+ /* portid = 0; this is *not* signifying the first port,
+ * rather, it signifies that portid is ignored.
+ */
+ ret = init_mem(0, NB_MBUF(nb_ports));
+ } else {
+ ret = init_mem(portid, NB_MBUF(1));
+ }
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "init_mem() failed\n");
+
+ /* Init one TX queue per couple (lcore,port) */
+ queueid = 0;
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+
+ qconf = &lcore_conf[lcore_id];
+
+ if (numa_on)
+ socketid = (uint8_t)rte_lcore_to_socket_id(
+ lcore_id);
+ else
+ socketid = 0;
+
+ printf("txq=%u,%d,%d ", lcore_id, queueid, socketid);
+ fflush(stdout);
+
+ txconf = &dev_info.default_txconf;
+ txconf->offloads = local_port_conf.txmode.offloads;
+ ret = rte_eth_tx_queue_setup(portid, queueid, nb_txd,
+ socketid, txconf);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "rte_eth_tx_queue_setup: err=%d, "
+ "port=%d\n",
+ ret, portid);
+ queueid++;
+ }
+
+ printf("\n");
+ }
+
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+ qconf = &lcore_conf[lcore_id];
+ printf("\nInitializing rx queues on lcore %u ... ", lcore_id);
+ fflush(stdout);
+ /* Init RX queues */
+ for (queue = 0; queue < qconf->n_rx_queue; ++queue) {
+ struct rte_eth_rxconf rxq_conf;
+
+ portid = qconf->rx_queue_list[queue].port_id;
+ queueid = qconf->rx_queue_list[queue].queue_id;
+
+ if (numa_on)
+ socketid = (uint8_t)rte_lcore_to_socket_id(
+ lcore_id);
+ else
+ socketid = 0;
+
+ printf("rxq=%d,%d,%d ", portid, queueid, socketid);
+ fflush(stdout);
+
+ rte_eth_dev_info_get(portid, &dev_info);
+ rxq_conf = dev_info.default_rxconf;
+ rxq_conf.offloads = port_conf.rxmode.offloads;
+ if (!per_port_pool)
+ ret = rte_eth_rx_queue_setup(
+ portid, queueid, nb_rxd, socketid,
+ &rxq_conf, pktmbuf_pool[0][socketid]);
+ else
+ ret = rte_eth_rx_queue_setup(
+ portid, queueid, nb_rxd, socketid,
+ &rxq_conf,
+ pktmbuf_pool[portid][socketid]);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "rte_eth_rx_queue_setup: err=%d, "
+ "port=%d\n",
+ ret, portid);
+
+ }
+ }
+
+ printf("\n");
+
+ /* Start ports */
+ RTE_ETH_FOREACH_DEV(portid)
+ {
+ if ((enabled_port_mask & (1 << portid)) == 0)
+ continue;
+
+ /* Start device */
+ ret = rte_eth_dev_start(portid);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "rte_eth_dev_start: err=%d, port=%d\n", ret,
+ portid);
+
+ /*
+ * If enabled, put device in promiscuous mode.
+ * This allows IO forwarding mode to forward packets
+ * to itself through 2 cross-connected ports of the
+ * target machine.
+ */
+ if (promiscuous_on)
+ rte_eth_promiscuous_enable(portid);
+ }
+
+ printf("\n");
+
+ check_all_ports_link_status(enabled_port_mask);
+
+ /* Stop ports */
+ RTE_ETH_FOREACH_DEV(portid) {
+ if ((enabled_port_mask & (1 << portid)) == 0)
+ continue;
+ printf("Closing port %d...", portid);
+ rte_eth_dev_stop(portid);
+ rte_eth_dev_close(portid);
+ printf(" Done\n");
+ }
printf("Bye...\n");
return ret;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v1 26/26] l3fwd-graph: add graph config and main loop
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (24 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 25/26] l3fwd-graph: add ethdev configuration changes jerinj
@ 2020-03-18 21:35 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
26 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-18 21:35 UTC (permalink / raw)
To: Marko Kovacevic, Ori Kam, Bruce Richardson, Radu Nicolau,
Akhil Goyal, Tomasz Kantecki, Sunil Kumar Kori, Pavan Nikhilesh,
Nithin Dabilpuram
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add graph creation, configuration logic and graph main loop.
This graph main loop is run on every slave lcore and calls
rte_graph_walk() to walk over lcore specific rte_graph.
Master core accumulates and prints graph walk stats of all the
lcore's graph's.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
examples/l3fwd-graph/main.c | 241 +++++++++++++++++++++++++++++++++++-
1 file changed, 239 insertions(+), 2 deletions(-)
diff --git a/examples/l3fwd-graph/main.c b/examples/l3fwd-graph/main.c
index 436987877..104a4d8db 100644
--- a/examples/l3fwd-graph/main.c
+++ b/examples/l3fwd-graph/main.c
@@ -22,9 +22,13 @@
#include <rte_cycles.h>
#include <rte_eal.h>
#include <rte_ethdev.h>
+#include <rte_graph_worker.h>
+#include <rte_launch.h>
#include <rte_lcore.h>
#include <rte_log.h>
#include <rte_mempool.h>
+#include <rte_node_eth_api.h>
+#include <rte_node_ip4_api.h>
#include <rte_per_lcore.h>
#include <rte_string_fns.h>
#include <rte_vect.h>
@@ -74,12 +78,17 @@ static uint32_t enabled_port_mask;
struct lcore_rx_queue {
uint16_t port_id;
uint8_t queue_id;
+ char node_name[RTE_NODE_NAMESIZE];
};
/* Lcore conf */
struct lcore_conf {
uint16_t n_rx_queue;
struct lcore_rx_queue rx_queue_list[MAX_RX_QUEUE_PER_LCORE];
+
+ struct rte_graph *graph;
+ char name[RTE_GRAPH_NAMESIZE];
+ rte_graph_t graph_id;
} __rte_cache_aligned;
static struct lcore_conf lcore_conf[RTE_MAX_LCORE];
@@ -118,6 +127,25 @@ static struct rte_eth_conf port_conf = {
static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
+static struct rte_node_ethdev_config ethdev_conf[RTE_MAX_ETHPORTS];
+
+struct ipv4_l3fwd_lpm_route {
+ uint32_t ip;
+ uint8_t depth;
+ uint8_t if_out;
+};
+
+#define IPV4_L3FWD_LPM_NUM_ROUTES \
+ (sizeof(ipv4_l3fwd_lpm_route_array) / \
+ sizeof(ipv4_l3fwd_lpm_route_array[0]))
+/* 198.18.0.0/16 are set aside for RFC2544 benchmarking. */
+static struct ipv4_l3fwd_lpm_route ipv4_l3fwd_lpm_route_array[] = {
+ {RTE_IPV4(198, 18, 0, 0), 24, 0}, {RTE_IPV4(198, 18, 1, 0), 24, 1},
+ {RTE_IPV4(198, 18, 2, 0), 24, 2}, {RTE_IPV4(198, 18, 3, 0), 24, 3},
+ {RTE_IPV4(198, 18, 4, 0), 24, 4}, {RTE_IPV4(198, 18, 5, 0), 24, 5},
+ {RTE_IPV4(198, 18, 6, 0), 24, 6}, {RTE_IPV4(198, 18, 7, 0), 24, 7},
+};
+
static int
check_lcore_params(void)
{
@@ -627,17 +655,87 @@ signal_handler(int signum)
}
}
+static void
+print_stats(void)
+{
+ const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'};
+ const char clr[] = {27, '[', '2', 'J', '\0'};
+ struct rte_graph_cluster_stats_param s_param;
+ struct rte_graph_cluster_stats *stats;
+ const char *pattern = "worker_*";
+
+ /* Prepare stats object */
+ memset(&s_param, 0, sizeof(s_param));
+ s_param.f = stdout;
+ s_param.socket_id = SOCKET_ID_ANY;
+ s_param.graph_patterns = &pattern;
+ s_param.nb_graph_patterns = 1;
+
+ stats = rte_graph_cluster_stats_create(&s_param);
+ if (stats == NULL)
+ rte_exit(EXIT_FAILURE, "Unable to create stats object\n");
+
+ while (!force_quit) {
+ /* Clear screen and move to top left */
+ printf("%s%s", clr, topLeft);
+ rte_graph_cluster_stats_get(stats, 0);
+ rte_delay_ms(1E3);
+ }
+
+ rte_graph_cluster_stats_destroy(stats);
+}
+
+/* Main processing loop */
+static int
+graph_main_loop(void *conf)
+{
+ struct lcore_conf *qconf;
+ struct rte_graph *graph;
+ uint32_t lcore_id;
+
+ RTE_SET_USED(conf);
+
+ lcore_id = rte_lcore_id();
+ qconf = &lcore_conf[lcore_id];
+ graph = qconf->graph;
+
+ if (!graph) {
+ RTE_LOG(INFO, L3FWD_GRAPH, "Lcore %u has nothing to do\n",
+ lcore_id);
+ return 0;
+ }
+
+ RTE_LOG(INFO, L3FWD_GRAPH,
+ "Entering main loop on lcore %u, graph %s(%p)\n", lcore_id,
+ qconf->name, graph);
+
+ while (likely(!force_quit))
+ rte_graph_walk(graph);
+
+ return 0;
+}
+
int
main(int argc, char **argv)
{
+ const char *node_patterns[64] = {
+ "ip4*",
+ "ethdev_tx-*",
+ "pkt_drop",
+ };
+ uint8_t rewrite_data[2 * sizeof(struct rte_ether_addr)];
uint8_t nb_rx_queue, queue, socketid;
+ struct rte_graph_param graph_conf;
struct rte_eth_dev_info dev_info;
+ uint32_t nb_ports, nb_conf = 0;
uint32_t n_tx_queue, nb_lcores;
struct rte_eth_txconf *txconf;
- uint16_t queueid, portid;
+ uint16_t queueid, portid, i;
struct lcore_conf *qconf;
+ uint16_t nb_patterns = 3;
+ uint16_t nb_graphs = 0;
+ uint8_t rewrite_len;
uint32_t lcore_id;
- uint32_t nb_ports;
int ret;
/* Init EAL */
@@ -786,6 +884,18 @@ main(int argc, char **argv)
queueid++;
}
+ /* Setup ethdev node config */
+ ethdev_conf[nb_conf].port_id = portid;
+ ethdev_conf[nb_conf].num_rx_queues = nb_rx_queue;
+ ethdev_conf[nb_conf].num_tx_queues = n_tx_queue;
+ if (!per_port_pool)
+ ethdev_conf[nb_conf].mp = pktmbuf_pool[0];
+
+ else
+ ethdev_conf[nb_conf].mp = pktmbuf_pool[portid];
+ ethdev_conf[nb_conf].mp_count = NB_SOCKETS;
+
+ nb_conf++;
printf("\n");
}
@@ -829,11 +939,26 @@ main(int argc, char **argv)
"port=%d\n",
ret, portid);
+ /* Add this queue node to its graph */
+ snprintf(qconf->rx_queue_list[queue].node_name,
+ RTE_NODE_NAMESIZE, "ethdev_rx-%u-%u", portid,
+ queueid);
+ }
+
+ /* Alloc a graph to this lcore only if source exists */
+ if (qconf->n_rx_queue) {
+ qconf->graph_id = nb_graphs;
+ nb_graphs++;
}
}
printf("\n");
+ /* Ethdev node config, skip rx queue mapping */
+ ret = rte_node_eth_config(ethdev_conf, nb_conf, nb_graphs);
+ if (ret)
+ rte_exit(EXIT_FAILURE, "rte_node_eth_config: err=%d\n", ret);
+
/* Start ports */
RTE_ETH_FOREACH_DEV(portid)
{
@@ -861,6 +986,118 @@ main(int argc, char **argv)
check_all_ports_link_status(enabled_port_mask);
+ /* Graph Initialization */
+ memset(&graph_conf, 0, sizeof(graph_conf));
+ graph_conf.node_patterns = node_patterns;
+
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ rte_graph_t graph_id;
+ rte_edge_t i;
+
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+
+ qconf = &lcore_conf[lcore_id];
+
+ /* Skip graph creation if no source exists */
+ if (!qconf->n_rx_queue)
+ continue;
+
+ /* Add rx node patterns of this lcore */
+ for (i = 0; i < qconf->n_rx_queue; i++) {
+ graph_conf.node_patterns[nb_patterns + i] =
+ qconf->rx_queue_list[i].node_name;
+ }
+
+ graph_conf.nb_node_patterns = nb_patterns + i;
+ graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id);
+
+ snprintf(qconf->name, sizeof(qconf->name), "worker_%u",
+ lcore_id);
+
+ graph_id = rte_graph_create(qconf->name, &graph_conf);
+ if (graph_id != qconf->graph_id)
+ rte_exit(EXIT_FAILURE,
+ "rte_graph_create(): graph_id=%d not "
+ " as expected for lcore %u(%u\n",
+ graph_id, lcore_id, qconf->graph_id);
+
+ qconf->graph = rte_graph_lookup(qconf->name);
+ if (!qconf->graph)
+ rte_exit(EXIT_FAILURE,
+ "rte_graph_lookup(): graph %s not found\n",
+ qconf->name);
+ }
+
+ memset(&rewrite_data, 0, sizeof(rewrite_data));
+ rewrite_len = sizeof(rewrite_data);
+
+ /* Add route to ip4 graph infra */
+ for (i = 0; i < IPV4_L3FWD_LPM_NUM_ROUTES; i++) {
+ char route_str[INET6_ADDRSTRLEN * 4];
+ char abuf[INET6_ADDRSTRLEN];
+ struct in_addr in;
+ uint32_t dst_port;
+ uint16_t next_hop;
+
+ /* Skip unused ports */
+ if ((1 << ipv4_l3fwd_lpm_route_array[i].if_out &
+ enabled_port_mask) == 0)
+ continue;
+
+ dst_port = ipv4_l3fwd_lpm_route_array[i].if_out;
+ next_hop = i;
+
+ in.s_addr = htonl(ipv4_l3fwd_lpm_route_array[i].ip);
+ snprintf(route_str, sizeof(route_str), "%s / %d (%d)",
+ inet_ntop(AF_INET, &in, abuf, sizeof(abuf)),
+ ipv4_l3fwd_lpm_route_array[i].depth,
+ ipv4_l3fwd_lpm_route_array[i].if_out);
+
+ ret = rte_node_ip4_route_add(
+ ipv4_l3fwd_lpm_route_array[i].ip,
+ ipv4_l3fwd_lpm_route_array[i].depth, next_hop,
+ RTE_NODE_IP4_LOOKUP_NEXT_REWRITE);
+
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "Unable to add ip4 route %s to graph\n",
+ route_str);
+
+ memcpy(rewrite_data, val_eth + dst_port, rewrite_len);
+
+ /* Add next hop for a given destination */
+ ret = rte_node_ip4_rewrite_add(next_hop, rewrite_data,
+ rewrite_len, dst_port);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "Unable to add next hop %u for "
+ "route %s\n",
+ next_hop, route_str);
+
+ RTE_LOG(INFO, L3FWD_GRAPH, "Added route %s, next_hop %u\n",
+ route_str, next_hop);
+ }
+
+ /* Launch per-lcore init on every slave lcore */
+ rte_eal_mp_remote_launch(graph_main_loop, NULL, SKIP_MASTER);
+
+ /* Accumulate and print stats on master until exit */
+ if (rte_graph_has_stats_feature())
+ print_stats();
+
+ /* Wait for slave cores to exit */
+ ret = 0;
+ RTE_LCORE_FOREACH_SLAVE(lcore_id) {
+ ret = rte_eal_wait_lcore(lcore_id);
+ /* Destroy graph */
+ rte_graph_destroy(lcore_conf[lcore_id].name);
+ if (ret < 0) {
+ ret = -1;
+ break;
+ }
+ }
+
/* Stop ports */
RTE_ETH_FOREACH_DEV(portid) {
if ((enabled_port_mask & (1 << portid)) == 0)
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup for x86 jerinj
@ 2020-03-19 12:25 ` Ray Kinsella
2020-03-19 14:22 ` [dpdk-dev] [EXT] " Pavan Nikhilesh Bhagavatula
0 siblings, 1 reply; 219+ messages in thread
From: Ray Kinsella @ 2020-03-19 12:25 UTC (permalink / raw)
To: jerinj, Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mattias.ronnblom, kirankumark
On 18/03/2020 21:35, jerinj@marvell.com wrote:
> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>
> Add IPv4 lookup process function for ip4_lookup
> rte_node. This node performs LPM lookup using x86_64
> vector supported RTE_LPM API on every packet received
> and forwards it to a next node that is identified by
> lookup result.
>
> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
> ---
> lib/librte_node/ip4_lookup.c | 245 +++++++++++++++++++++++++++++++++++
> 1 file changed, 245 insertions(+)
>
> diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c
> index d7fcd1158..c003e9c91 100644
> --- a/lib/librte_node/ip4_lookup.c
> +++ b/lib/librte_node/ip4_lookup.c
> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
> return nb_objs;
> }
>
> +#elif defined(RTE_ARCH_X86)
> +
> +/* X86 SSE */
> +static uint16_t
> +ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
> + void **objs, uint16_t nb_objs)
> +{
> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
> + rte_edge_t next0, next1, next2, next3, next_index;
> + struct rte_ipv4_hdr *ipv4_hdr;
> + struct rte_ether_hdr *eth_hdr;
> + uint32_t ip0, ip1, ip2, ip3;
> + void **to_next, **from;
> + uint16_t last_spec = 0;
> + uint16_t n_left_from;
> + struct rte_lpm *lpm;
> + uint16_t held = 0;
> + uint32_t drop_nh;
> + rte_xmm_t dst;
> + __m128i dip; /* SSE register */
> + int rc, i;
> +
> + /* Speculative next */
> + next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
> + /* Drop node */
> + drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
> +
> + /* Get socket specific LPM from ctx */
> + lpm = *((struct rte_lpm **)node->ctx);
> +
> + pkts = (struct rte_mbuf **)objs;
> + from = objs;
> + n_left_from = nb_objs;
I doubt this initial prefetch of the first 4 packets has any benefit.
> + if (n_left_from >= 4) {
> + for (i = 0; i < 4; i++) {
> + rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
> + struct rte_ether_hdr *) +
> + 1);
> + }
> + }
> +
> + /* Get stream for the speculated next node */
> + to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
Suggest you don't reuse the hand-unrolling optimization from FD.io VPP.
I have never found any performance benefit from them, and they make the code unnecessarily verbose.
> + while (n_left_from >= 4) {
> + /* Prefetch next-next mbufs */
> + if (likely(n_left_from >= 11)) {
> + rte_prefetch0(pkts[8]);
> + rte_prefetch0(pkts[9]);
> + rte_prefetch0(pkts[10]);
> + rte_prefetch0(pkts[11]);
> + }
> +
> + /* Prefetch next mbuf data */
> + if (likely(n_left_from >= 7)) {
> + rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
> + struct rte_ether_hdr *) +
> + 1);
> + rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
> + struct rte_ether_hdr *) +
> + 1);
> + rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
> + struct rte_ether_hdr *) +
> + 1);
> + rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
> + struct rte_ether_hdr *) +
> + 1);
> + }
> +
> + mbuf0 = pkts[0];
> + mbuf1 = pkts[1];
> + mbuf2 = pkts[2];
> + mbuf3 = pkts[3];
> +
> + pkts += 4;
> + n_left_from -= 4;
> +
> + /* Extract DIP of mbuf0 */
> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
> + ip0 = ipv4_hdr->dst_addr;
> + /* Extract cksum, ttl as ipv4 hdr is in cache */
> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
> +
> + /* Extract DIP of mbuf1 */
> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct rte_ether_hdr *);
> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
> + ip1 = ipv4_hdr->dst_addr;
> + /* Extract cksum, ttl as ipv4 hdr is in cache */
> + rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr->hdr_checksum;
> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr->time_to_live;
> +
> + /* Extract DIP of mbuf2 */
> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct rte_ether_hdr *);
> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
> + ip2 = ipv4_hdr->dst_addr;
> + /* Extract cksum, ttl as ipv4 hdr is in cache */
> + rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr->hdr_checksum;
> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr->time_to_live;
> +
> + /* Extract DIP of mbuf3 */
> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct rte_ether_hdr *);
> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
> + ip3 = ipv4_hdr->dst_addr;
> +
> + /* Prepare for lookup x4 */
> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
> +
> + /* Byte swap 4 IPV4 addresses. */
> + const __m128i bswap_mask = _mm_set_epi8(
> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
> + dip = _mm_shuffle_epi8(dip, bswap_mask);
> +
> + /* Extract cksum, ttl as ipv4 hdr is in cache */
> + rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr->hdr_checksum;
> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr->time_to_live;
> +
> + /* Perform LPM lookup to get NH and next node */
> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
> +
> + /* Extract next node id and NH */
> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] & 0xFFFF;
> + next0 = (dst.u32[0] >> 16);
> +
> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] & 0xFFFF;
> + next1 = (dst.u32[1] >> 16);
> +
> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] & 0xFFFF;
> + next2 = (dst.u32[2] >> 16);
> +
> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] & 0xFFFF;
> + next3 = (dst.u32[3] >> 16);
> +
> + /* Enqueue four to next node */
> + rte_edge_t fix_spec =
> + (next_index ^ next0) | (next_index ^ next1) |
> + (next_index ^ next2) | (next_index ^ next3);
> +
> + if (unlikely(fix_spec)) {
> + /* Copy things successfully speculated till now */
> + rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
> + from += last_spec;
> + to_next += last_spec;
> + held += last_spec;
> + last_spec = 0;
> +
> + /* Next0 */
> + if (next_index == next0) {
> + to_next[0] = from[0];
> + to_next++;
> + held++;
> + } else {
> + rte_node_enqueue_x1(graph, node, next0,
> + from[0]);
> + }
> +
> + /* Next1 */
> + if (next_index == next1) {
> + to_next[0] = from[1];
> + to_next++;
> + held++;
> + } else {
> + rte_node_enqueue_x1(graph, node, next1,
> + from[1]);
> + }
> +
> + /* Next2 */
> + if (next_index == next2) {
> + to_next[0] = from[2];
> + to_next++;
> + held++;
> + } else {
> + rte_node_enqueue_x1(graph, node, next2,
> + from[2]);
> + }
> +
> + /* Next3 */
> + if (next_index == next3) {
> + to_next[0] = from[3];
> + to_next++;
> + held++;
> + } else {
> + rte_node_enqueue_x1(graph, node, next3,
> + from[3]);
> + }
> +
> + from += 4;
> +
> + } else {
> + last_spec += 4;
> + }
> + }
> +
> + while (n_left_from > 0) {
> + uint32_t next_hop;
> +
> + mbuf0 = pkts[0];
> +
> + pkts += 1;
> + n_left_from -= 1;
> +
> + /* Extract DIP of mbuf0 */
> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
> + /* Extract cksum, ttl as ipv4 hdr is in cache */
> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
> +
> + rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
> + &next_hop);
> + next_hop = (rc == 0) ? next_hop : drop_nh;
> +
> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop & 0xFFFF;
> + next0 = (next_hop >> 16);
> +
> + if (unlikely(next_index ^ next0)) {
> + /* Copy things successfully speculated till now */
> + rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
> + from += last_spec;
> + to_next += last_spec;
> + held += last_spec;
> + last_spec = 0;
> +
> + rte_node_enqueue_x1(graph, node, next0, from[0]);
> + from += 1;
> + } else {
> + last_spec += 1;
> + }
> + }
> +
> + /* !!! Home run !!! */
> + if (likely(last_spec == nb_objs)) {
> + rte_node_next_stream_move(graph, node, next_index);
> + return nb_objs;
> + }
> +
> + held += last_spec;
> + /* Copy things successfully speculated till now */
> + rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
> + rte_node_next_stream_put(graph, node, next_index, held);
> +
> + return nb_objs;
> +}
> +
> #else
>
> static uint16_t
>
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-19 12:25 ` Ray Kinsella
@ 2020-03-19 14:22 ` Pavan Nikhilesh Bhagavatula
2020-03-19 15:50 ` Ray Kinsella
0 siblings, 1 reply; 219+ messages in thread
From: Pavan Nikhilesh Bhagavatula @ 2020-03-19 14:22 UTC (permalink / raw)
To: Ray Kinsella, Jerin Jacob Kollanukkaran, Nithin Kumar Dabilpuram
Cc: dev, thomas, david.marchand, mattias.ronnblom, Kiran Kumar Kokkilagadda
>On 18/03/2020 21:35, jerinj@marvell.com wrote:
>> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>
>> Add IPv4 lookup process function for ip4_lookup
>> rte_node. This node performs LPM lookup using x86_64
>> vector supported RTE_LPM API on every packet received
>> and forwards it to a next node that is identified by
>> lookup result.
>>
>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
>> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
>> ---
>> lib/librte_node/ip4_lookup.c | 245
>+++++++++++++++++++++++++++++++++++
>> 1 file changed, 245 insertions(+)
>>
>> diff --git a/lib/librte_node/ip4_lookup.c
>b/lib/librte_node/ip4_lookup.c
>> index d7fcd1158..c003e9c91 100644
>> --- a/lib/librte_node/ip4_lookup.c
>> +++ b/lib/librte_node/ip4_lookup.c
>> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct rte_graph
>*graph, struct rte_node *node,
>> return nb_objs;
>> }
>>
>> +#elif defined(RTE_ARCH_X86)
>> +
>> +/* X86 SSE */
>> +static uint16_t
>> +ip4_lookup_node_process(struct rte_graph *graph, struct rte_node
>*node,
>> + void **objs, uint16_t nb_objs)
>> +{
>> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
>> + rte_edge_t next0, next1, next2, next3, next_index;
>> + struct rte_ipv4_hdr *ipv4_hdr;
>> + struct rte_ether_hdr *eth_hdr;
>> + uint32_t ip0, ip1, ip2, ip3;
>> + void **to_next, **from;
>> + uint16_t last_spec = 0;
>> + uint16_t n_left_from;
>> + struct rte_lpm *lpm;
>> + uint16_t held = 0;
>> + uint32_t drop_nh;
>> + rte_xmm_t dst;
>> + __m128i dip; /* SSE register */
>> + int rc, i;
>> +
>> + /* Speculative next */
>> + next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
>> + /* Drop node */
>> + drop_nh =
>((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
>> +
>> + /* Get socket specific LPM from ctx */
>> + lpm = *((struct rte_lpm **)node->ctx);
>> +
>> + pkts = (struct rte_mbuf **)objs;
>> + from = objs;
>> + n_left_from = nb_objs;
>
>I doubt this initial prefetch of the first 4 packets has any benefit.
Ack will remove in v2 for x86.
>
>> + if (n_left_from >= 4) {
>> + for (i = 0; i < 4; i++) {
>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
>> + struct rte_ether_hdr
>*) +
>> + 1);
>> + }
>> + }
>> +
>> + /* Get stream for the speculated next node */
>> + to_next = rte_node_next_stream_get(graph, node,
>next_index, nb_objs);
>
>Suggest you don't reuse the hand-unrolling optimization from FD.io
>VPP.
>I have never found any performance benefit from them, and they
>make the code unnecessarily verbose.
>
How would be take the benefit of rte_lpm_lookupx4 without unrolling the loop?.
Also, in future if we are using rte_rib and fib with a CPU supporting wider SIMD we might
need to unroll them further (AVX256 AND 512 currently rte_lpm_lookup uses only 128bit
since it is only uses SSE extension).
>
>> + while (n_left_from >= 4) {
>> + /* Prefetch next-next mbufs */
>> + if (likely(n_left_from >= 11)) {
>> + rte_prefetch0(pkts[8]);
>> + rte_prefetch0(pkts[9]);
>> + rte_prefetch0(pkts[10]);
>> + rte_prefetch0(pkts[11]);
>> + }
>> +
>> + /* Prefetch next mbuf data */
>> + if (likely(n_left_from >= 7)) {
>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
>> + struct rte_ether_hdr
>*) +
>> + 1);
>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
>> + struct rte_ether_hdr
>*) +
>> + 1);
>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
>> + struct rte_ether_hdr
>*) +
>> + 1);
>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
>> + struct rte_ether_hdr
>*) +
>> + 1);
>> + }
>> +
>> + mbuf0 = pkts[0];
>> + mbuf1 = pkts[1];
>> + mbuf2 = pkts[2];
>> + mbuf3 = pkts[3];
>> +
>> + pkts += 4;
>> + n_left_from -= 4;
>> +
>> + /* Extract DIP of mbuf0 */
>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>rte_ether_hdr *);
>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>> + ip0 = ipv4_hdr->dst_addr;
>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
>>hdr_checksum;
>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>time_to_live;
>> +
>> + /* Extract DIP of mbuf1 */
>> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct
>rte_ether_hdr *);
>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>> + ip1 = ipv4_hdr->dst_addr;
>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>> + rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr-
>>hdr_checksum;
>> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-
>>time_to_live;
>> +
>> + /* Extract DIP of mbuf2 */
>> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct
>rte_ether_hdr *);
>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>> + ip2 = ipv4_hdr->dst_addr;
>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>> + rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr-
>>hdr_checksum;
>> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-
>>time_to_live;
>> +
>> + /* Extract DIP of mbuf3 */
>> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct
>rte_ether_hdr *);
>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>> + ip3 = ipv4_hdr->dst_addr;
>> +
>> + /* Prepare for lookup x4 */
>> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
>> +
>> + /* Byte swap 4 IPV4 addresses. */
>> + const __m128i bswap_mask = _mm_set_epi8(
>> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
>> + dip = _mm_shuffle_epi8(dip, bswap_mask);
>> +
>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>> + rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr-
>>hdr_checksum;
>> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-
>>time_to_live;
>> +
>> + /* Perform LPM lookup to get NH and next node */
>> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
>> +
>> + /* Extract next node id and NH */
>> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] &
>0xFFFF;
>> + next0 = (dst.u32[0] >> 16);
>> +
>> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] &
>0xFFFF;
>> + next1 = (dst.u32[1] >> 16);
>> +
>> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] &
>0xFFFF;
>> + next2 = (dst.u32[2] >> 16);
>> +
>> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] &
>0xFFFF;
>> + next3 = (dst.u32[3] >> 16);
>> +
>> + /* Enqueue four to next node */
>> + rte_edge_t fix_spec =
>> + (next_index ^ next0) | (next_index ^ next1) |
>> + (next_index ^ next2) | (next_index ^ next3);
>> +
>> + if (unlikely(fix_spec)) {
>> + /* Copy things successfully speculated till now
>*/
>> + rte_memcpy(to_next, from, last_spec *
>sizeof(from[0]));
>> + from += last_spec;
>> + to_next += last_spec;
>> + held += last_spec;
>> + last_spec = 0;
>> +
>> + /* Next0 */
>> + if (next_index == next0) {
>> + to_next[0] = from[0];
>> + to_next++;
>> + held++;
>> + } else {
>> + rte_node_enqueue_x1(graph, node,
>next0,
>> + from[0]);
>> + }
>> +
>> + /* Next1 */
>> + if (next_index == next1) {
>> + to_next[0] = from[1];
>> + to_next++;
>> + held++;
>> + } else {
>> + rte_node_enqueue_x1(graph, node,
>next1,
>> + from[1]);
>> + }
>> +
>> + /* Next2 */
>> + if (next_index == next2) {
>> + to_next[0] = from[2];
>> + to_next++;
>> + held++;
>> + } else {
>> + rte_node_enqueue_x1(graph, node,
>next2,
>> + from[2]);
>> + }
>> +
>> + /* Next3 */
>> + if (next_index == next3) {
>> + to_next[0] = from[3];
>> + to_next++;
>> + held++;
>> + } else {
>> + rte_node_enqueue_x1(graph, node,
>next3,
>> + from[3]);
>> + }
>> +
>> + from += 4;
>> +
>> + } else {
>> + last_spec += 4;
>> + }
>> + }
>> +
>> + while (n_left_from > 0) {
>> + uint32_t next_hop;
>> +
>> + mbuf0 = pkts[0];
>> +
>> + pkts += 1;
>> + n_left_from -= 1;
>> +
>> + /* Extract DIP of mbuf0 */
>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>rte_ether_hdr *);
>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
>>hdr_checksum;
>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>time_to_live;
>> +
>> + rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr-
>>dst_addr),
>> + &next_hop);
>> + next_hop = (rc == 0) ? next_hop : drop_nh;
>> +
>> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop &
>0xFFFF;
>> + next0 = (next_hop >> 16);
>> +
>> + if (unlikely(next_index ^ next0)) {
>> + /* Copy things successfully speculated till now
>*/
>> + rte_memcpy(to_next, from, last_spec *
>sizeof(from[0]));
>> + from += last_spec;
>> + to_next += last_spec;
>> + held += last_spec;
>> + last_spec = 0;
>> +
>> + rte_node_enqueue_x1(graph, node, next0,
>from[0]);
>> + from += 1;
>> + } else {
>> + last_spec += 1;
>> + }
>> + }
>> +
>> + /* !!! Home run !!! */
>> + if (likely(last_spec == nb_objs)) {
>> + rte_node_next_stream_move(graph, node,
>next_index);
>> + return nb_objs;
>> + }
>> +
>> + held += last_spec;
>> + /* Copy things successfully speculated till now */
>> + rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
>> + rte_node_next_stream_put(graph, node, next_index, held);
>> +
>> + return nb_objs;
>> +}
>> +
>> #else
>>
>> static uint16_t
>>
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-19 14:22 ` [dpdk-dev] [EXT] " Pavan Nikhilesh Bhagavatula
@ 2020-03-19 15:50 ` Ray Kinsella
2020-03-19 16:13 ` Pavan Nikhilesh Bhagavatula
0 siblings, 1 reply; 219+ messages in thread
From: Ray Kinsella @ 2020-03-19 15:50 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula, Jerin Jacob Kollanukkaran,
Nithin Kumar Dabilpuram
Cc: dev, thomas, david.marchand, mattias.ronnblom, Kiran Kumar Kokkilagadda
On 19/03/2020 14:22, Pavan Nikhilesh Bhagavatula wrote:
>> On 18/03/2020 21:35, jerinj@marvell.com wrote:
>>> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>
>>> Add IPv4 lookup process function for ip4_lookup
>>> rte_node. This node performs LPM lookup using x86_64
>>> vector supported RTE_LPM API on every packet received
>>> and forwards it to a next node that is identified by
>>> lookup result.
>>>
>>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
>>> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
>>> ---
>>> lib/librte_node/ip4_lookup.c | 245
>> +++++++++++++++++++++++++++++++++++
>>> 1 file changed, 245 insertions(+)
>>>
>>> diff --git a/lib/librte_node/ip4_lookup.c
>> b/lib/librte_node/ip4_lookup.c
>>> index d7fcd1158..c003e9c91 100644
>>> --- a/lib/librte_node/ip4_lookup.c
>>> +++ b/lib/librte_node/ip4_lookup.c
>>> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct rte_graph
>> *graph, struct rte_node *node,
>>> return nb_objs;
>>> }
>>>
>>> +#elif defined(RTE_ARCH_X86)
>>> +
>>> +/* X86 SSE */
>>> +static uint16_t
>>> +ip4_lookup_node_process(struct rte_graph *graph, struct rte_node
>> *node,
>>> + void **objs, uint16_t nb_objs)
>>> +{
>>> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
>>> + rte_edge_t next0, next1, next2, next3, next_index;
>>> + struct rte_ipv4_hdr *ipv4_hdr;
>>> + struct rte_ether_hdr *eth_hdr;
>>> + uint32_t ip0, ip1, ip2, ip3;
>>> + void **to_next, **from;
>>> + uint16_t last_spec = 0;
>>> + uint16_t n_left_from;
>>> + struct rte_lpm *lpm;
>>> + uint16_t held = 0;
>>> + uint32_t drop_nh;
>>> + rte_xmm_t dst;
>>> + __m128i dip; /* SSE register */
>>> + int rc, i;
>>> +
>>> + /* Speculative next */
>>> + next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
>>> + /* Drop node */
>>> + drop_nh =
>> ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
>>> +
>>> + /* Get socket specific LPM from ctx */
>>> + lpm = *((struct rte_lpm **)node->ctx);
>>> +
>>> + pkts = (struct rte_mbuf **)objs;
>>> + from = objs;
>>> + n_left_from = nb_objs;
>>
>> I doubt this initial prefetch of the first 4 packets has any benefit.
>
> Ack will remove in v2 for x86.
>
>>
>>> + if (n_left_from >= 4) {
>>> + for (i = 0; i < 4; i++) {
>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
>>> + struct rte_ether_hdr
>> *) +
>>> + 1);
>>> + }
>>> + }
>>> +
>>> + /* Get stream for the speculated next node */
>>> + to_next = rte_node_next_stream_get(graph, node,
>> next_index, nb_objs);
>>
>> Suggest you don't reuse the hand-unrolling optimization from FD.io
>> VPP.
>> I have never found any performance benefit from them, and they
>> make the code unnecessarily verbose.
>>
>
> How would be take the benefit of rte_lpm_lookupx4 without unrolling the loop?.
> Also, in future if we are using rte_rib and fib with a CPU supporting wider SIMD we might
> need to unroll them further (AVX256 AND 512 currently rte_lpm_lookup uses only 128bit
> since it is only uses SSE extension).
Let the compiler do it for you, but using a constant vector length.
for (int i=0; i < 4; ++i) { ... }
>
>>
>>> + while (n_left_from >= 4) {
>>> + /* Prefetch next-next mbufs */
>>> + if (likely(n_left_from >= 11)) {
>>> + rte_prefetch0(pkts[8]);
>>> + rte_prefetch0(pkts[9]);
>>> + rte_prefetch0(pkts[10]);
>>> + rte_prefetch0(pkts[11]);
>>> + }
>>> +
>>> + /* Prefetch next mbuf data */
>>> + if (likely(n_left_from >= 7)) {
>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
>>> + struct rte_ether_hdr
>> *) +
>>> + 1);
>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
>>> + struct rte_ether_hdr
>> *) +
>>> + 1);
>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
>>> + struct rte_ether_hdr
>> *) +
>>> + 1);
>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
>>> + struct rte_ether_hdr
>> *) +
>>> + 1);
>>> + }
>>> +
>>> + mbuf0 = pkts[0];
>>> + mbuf1 = pkts[1];
>>> + mbuf2 = pkts[2];
>>> + mbuf3 = pkts[3];
>>> +
>>> + pkts += 4;
>>> + n_left_from -= 4;
>>> +
>>> + /* Extract DIP of mbuf0 */
>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>> rte_ether_hdr *);
>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>> + ip0 = ipv4_hdr->dst_addr;
>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
>>> hdr_checksum;
>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>> time_to_live;
>>> +
>>> + /* Extract DIP of mbuf1 */
>>> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct
>> rte_ether_hdr *);
>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>> + ip1 = ipv4_hdr->dst_addr;
>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>> + rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr-
>>> hdr_checksum;
>>> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-
>>> time_to_live;
>>> +
>>> + /* Extract DIP of mbuf2 */
>>> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct
>> rte_ether_hdr *);
>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>> + ip2 = ipv4_hdr->dst_addr;
>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>> + rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr-
>>> hdr_checksum;
>>> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-
>>> time_to_live;
>>> +
>>> + /* Extract DIP of mbuf3 */
>>> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct
>> rte_ether_hdr *);
>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>> + ip3 = ipv4_hdr->dst_addr;
>>> +
>>> + /* Prepare for lookup x4 */
>>> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
>>> +
>>> + /* Byte swap 4 IPV4 addresses. */
>>> + const __m128i bswap_mask = _mm_set_epi8(
>>> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
>>> + dip = _mm_shuffle_epi8(dip, bswap_mask);
>>> +
>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>> + rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr-
>>> hdr_checksum;
>>> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-
>>> time_to_live;
>>> +
>>> + /* Perform LPM lookup to get NH and next node */
>>> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
>>> +
>>> + /* Extract next node id and NH */
>>> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] &
>> 0xFFFF;
>>> + next0 = (dst.u32[0] >> 16);
>>> +
>>> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] &
>> 0xFFFF;
>>> + next1 = (dst.u32[1] >> 16);
>>> +
>>> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] &
>> 0xFFFF;
>>> + next2 = (dst.u32[2] >> 16);
>>> +
>>> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] &
>> 0xFFFF;
>>> + next3 = (dst.u32[3] >> 16);
>>> +
>>> + /* Enqueue four to next node */
>>> + rte_edge_t fix_spec =
>>> + (next_index ^ next0) | (next_index ^ next1) |
>>> + (next_index ^ next2) | (next_index ^ next3);
>>> +
>>> + if (unlikely(fix_spec)) {
>>> + /* Copy things successfully speculated till now
>> */
>>> + rte_memcpy(to_next, from, last_spec *
>> sizeof(from[0]));
>>> + from += last_spec;
>>> + to_next += last_spec;
>>> + held += last_spec;
>>> + last_spec = 0;
>>> +
>>> + /* Next0 */
>>> + if (next_index == next0) {
>>> + to_next[0] = from[0];
>>> + to_next++;
>>> + held++;
>>> + } else {
>>> + rte_node_enqueue_x1(graph, node,
>> next0,
>>> + from[0]);
>>> + }
>>> +
>>> + /* Next1 */
>>> + if (next_index == next1) {
>>> + to_next[0] = from[1];
>>> + to_next++;
>>> + held++;
>>> + } else {
>>> + rte_node_enqueue_x1(graph, node,
>> next1,
>>> + from[1]);
>>> + }
>>> +
>>> + /* Next2 */
>>> + if (next_index == next2) {
>>> + to_next[0] = from[2];
>>> + to_next++;
>>> + held++;
>>> + } else {
>>> + rte_node_enqueue_x1(graph, node,
>> next2,
>>> + from[2]);
>>> + }
>>> +
>>> + /* Next3 */
>>> + if (next_index == next3) {
>>> + to_next[0] = from[3];
>>> + to_next++;
>>> + held++;
>>> + } else {
>>> + rte_node_enqueue_x1(graph, node,
>> next3,
>>> + from[3]);
>>> + }
>>> +
>>> + from += 4;
>>> +
>>> + } else {
>>> + last_spec += 4;
>>> + }
>>> + }
>>> +
>>> + while (n_left_from > 0) {
>>> + uint32_t next_hop;
>>> +
>>> + mbuf0 = pkts[0];
>>> +
>>> + pkts += 1;
>>> + n_left_from -= 1;
>>> +
>>> + /* Extract DIP of mbuf0 */
>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>> rte_ether_hdr *);
>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
>>> hdr_checksum;
>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>> time_to_live;
>>> +
>>> + rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr-
>>> dst_addr),
>>> + &next_hop);
>>> + next_hop = (rc == 0) ? next_hop : drop_nh;
>>> +
>>> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop &
>> 0xFFFF;
>>> + next0 = (next_hop >> 16);
>>> +
>>> + if (unlikely(next_index ^ next0)) {
>>> + /* Copy things successfully speculated till now
>> */
>>> + rte_memcpy(to_next, from, last_spec *
>> sizeof(from[0]));
>>> + from += last_spec;
>>> + to_next += last_spec;
>>> + held += last_spec;
>>> + last_spec = 0;
>>> +
>>> + rte_node_enqueue_x1(graph, node, next0,
>> from[0]);
>>> + from += 1;
>>> + } else {
>>> + last_spec += 1;
>>> + }
>>> + }
>>> +
>>> + /* !!! Home run !!! */
>>> + if (likely(last_spec == nb_objs)) {
>>> + rte_node_next_stream_move(graph, node,
>> next_index);
>>> + return nb_objs;
>>> + }
>>> +
>>> + held += last_spec;
>>> + /* Copy things successfully speculated till now */
>>> + rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
>>> + rte_node_next_stream_put(graph, node, next_index, held);
>>> +
>>> + return nb_objs;
>>> +}
>>> +
>>> #else
>>>
>>> static uint16_t
>>>
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-19 15:50 ` Ray Kinsella
@ 2020-03-19 16:13 ` Pavan Nikhilesh Bhagavatula
2020-03-20 9:14 ` Ray Kinsella
0 siblings, 1 reply; 219+ messages in thread
From: Pavan Nikhilesh Bhagavatula @ 2020-03-19 16:13 UTC (permalink / raw)
To: Ray Kinsella, Jerin Jacob Kollanukkaran, Nithin Kumar Dabilpuram
Cc: dev, thomas, david.marchand, mattias.ronnblom, Kiran Kumar Kokkilagadda
>-----Original Message-----
>From: Ray Kinsella <mdr@ashroe.eu>
>Sent: Thursday, March 19, 2020 9:21 PM
>To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Jerin
>Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar Dabilpuram
><ndabilpuram@marvell.com>
>Cc: dev@dpdk.org; thomas@monjalon.net;
>david.marchand@redhat.com; mattias.ronnblom@ericsson.com; Kiran
>Kumar Kokkilagadda <kirankumark@marvell.com>
>Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup
>for x86
>
>
>
>On 19/03/2020 14:22, Pavan Nikhilesh Bhagavatula wrote:
>>> On 18/03/2020 21:35, jerinj@marvell.com wrote:
>>>> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>
>>>> Add IPv4 lookup process function for ip4_lookup
>>>> rte_node. This node performs LPM lookup using x86_64
>>>> vector supported RTE_LPM API on every packet received
>>>> and forwards it to a next node that is identified by
>>>> lookup result.
>>>>
>>>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
>>>> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
>>>> ---
>>>> lib/librte_node/ip4_lookup.c | 245
>>> +++++++++++++++++++++++++++++++++++
>>>> 1 file changed, 245 insertions(+)
>>>>
>>>> diff --git a/lib/librte_node/ip4_lookup.c
>>> b/lib/librte_node/ip4_lookup.c
>>>> index d7fcd1158..c003e9c91 100644
>>>> --- a/lib/librte_node/ip4_lookup.c
>>>> +++ b/lib/librte_node/ip4_lookup.c
>>>> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct
>rte_graph
>>> *graph, struct rte_node *node,
>>>> return nb_objs;
>>>> }
>>>>
>>>> +#elif defined(RTE_ARCH_X86)
>>>> +
>>>> +/* X86 SSE */
>>>> +static uint16_t
>>>> +ip4_lookup_node_process(struct rte_graph *graph, struct
>rte_node
>>> *node,
>>>> + void **objs, uint16_t nb_objs)
>>>> +{
>>>> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
>>>> + rte_edge_t next0, next1, next2, next3, next_index;
>>>> + struct rte_ipv4_hdr *ipv4_hdr;
>>>> + struct rte_ether_hdr *eth_hdr;
>>>> + uint32_t ip0, ip1, ip2, ip3;
>>>> + void **to_next, **from;
>>>> + uint16_t last_spec = 0;
>>>> + uint16_t n_left_from;
>>>> + struct rte_lpm *lpm;
>>>> + uint16_t held = 0;
>>>> + uint32_t drop_nh;
>>>> + rte_xmm_t dst;
>>>> + __m128i dip; /* SSE register */
>>>> + int rc, i;
>>>> +
>>>> + /* Speculative next */
>>>> + next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
>>>> + /* Drop node */
>>>> + drop_nh =
>>> ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
>>>> +
>>>> + /* Get socket specific LPM from ctx */
>>>> + lpm = *((struct rte_lpm **)node->ctx);
>>>> +
>>>> + pkts = (struct rte_mbuf **)objs;
>>>> + from = objs;
>>>> + n_left_from = nb_objs;
>>>
>>> I doubt this initial prefetch of the first 4 packets has any benefit.
>>
>> Ack will remove in v2 for x86.
>>
>>>
>>>> + if (n_left_from >= 4) {
>>>> + for (i = 0; i < 4; i++) {
>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
>>>> + struct rte_ether_hdr
>>> *) +
>>>> + 1);
>>>> + }
>>>> + }
>>>> +
>>>> + /* Get stream for the speculated next node */
>>>> + to_next = rte_node_next_stream_get(graph, node,
>>> next_index, nb_objs);
>>>
>>> Suggest you don't reuse the hand-unrolling optimization from FD.io
>>> VPP.
>>> I have never found any performance benefit from them, and they
>>> make the code unnecessarily verbose.
>>>
>>
>> How would be take the benefit of rte_lpm_lookupx4 without
>unrolling the loop?.
>> Also, in future if we are using rte_rib and fib with a CPU supporting
>wider SIMD we might
>> need to unroll them further (AVX256 AND 512 currently
>rte_lpm_lookup uses only 128bit
>> since it is only uses SSE extension).
>
>Let the compiler do it for you, but using a constant vector length.
>for (int i=0; i < 4; ++i) { ... }
>
Ok, I think I misunderstood the previous comment.
It was only for the prefetches in the loop right?
>>
>>>
>>>> + while (n_left_from >= 4) {
>>>> + /* Prefetch next-next mbufs */
>>>> + if (likely(n_left_from >= 11)) {
>>>> + rte_prefetch0(pkts[8]);
>>>> + rte_prefetch0(pkts[9]);
>>>> + rte_prefetch0(pkts[10]);
>>>> + rte_prefetch0(pkts[11]);
>>>> + }
>>>> +
>>>> + /* Prefetch next mbuf data */
>>>> + if (likely(n_left_from >= 7)) {
>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
>>>> + struct rte_ether_hdr
>>> *) +
>>>> + 1);
>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
>>>> + struct rte_ether_hdr
>>> *) +
>>>> + 1);
>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
>>>> + struct rte_ether_hdr
>>> *) +
>>>> + 1);
>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
>>>> + struct rte_ether_hdr
>>> *) +
>>>> + 1);
>>>> + }
>>>> +
>>>> + mbuf0 = pkts[0];
>>>> + mbuf1 = pkts[1];
>>>> + mbuf2 = pkts[2];
>>>> + mbuf3 = pkts[3];
>>>> +
>>>> + pkts += 4;
>>>> + n_left_from -= 4;
>>>> +
>>>> + /* Extract DIP of mbuf0 */
>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>> rte_ether_hdr *);
>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>> + ip0 = ipv4_hdr->dst_addr;
>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
>>>> hdr_checksum;
>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>> time_to_live;
>>>> +
>>>> + /* Extract DIP of mbuf1 */
>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct
>>> rte_ether_hdr *);
>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>> + ip1 = ipv4_hdr->dst_addr;
>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>> + rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr-
>>>> hdr_checksum;
>>>> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-
>>>> time_to_live;
>>>> +
>>>> + /* Extract DIP of mbuf2 */
>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct
>>> rte_ether_hdr *);
>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>> + ip2 = ipv4_hdr->dst_addr;
>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>> + rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr-
>>>> hdr_checksum;
>>>> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-
>>>> time_to_live;
>>>> +
>>>> + /* Extract DIP of mbuf3 */
>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct
>>> rte_ether_hdr *);
>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>> + ip3 = ipv4_hdr->dst_addr;
>>>> +
>>>> + /* Prepare for lookup x4 */
>>>> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
>>>> +
>>>> + /* Byte swap 4 IPV4 addresses. */
>>>> + const __m128i bswap_mask = _mm_set_epi8(
>>>> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
>>>> + dip = _mm_shuffle_epi8(dip, bswap_mask);
>>>> +
>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>> + rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr-
>>>> hdr_checksum;
>>>> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-
>>>> time_to_live;
>>>> +
>>>> + /* Perform LPM lookup to get NH and next node */
>>>> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
>>>> +
>>>> + /* Extract next node id and NH */
>>>> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] &
>>> 0xFFFF;
>>>> + next0 = (dst.u32[0] >> 16);
>>>> +
>>>> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] &
>>> 0xFFFF;
>>>> + next1 = (dst.u32[1] >> 16);
>>>> +
>>>> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] &
>>> 0xFFFF;
>>>> + next2 = (dst.u32[2] >> 16);
>>>> +
>>>> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] &
>>> 0xFFFF;
>>>> + next3 = (dst.u32[3] >> 16);
>>>> +
>>>> + /* Enqueue four to next node */
>>>> + rte_edge_t fix_spec =
>>>> + (next_index ^ next0) | (next_index ^ next1) |
>>>> + (next_index ^ next2) | (next_index ^ next3);
>>>> +
>>>> + if (unlikely(fix_spec)) {
>>>> + /* Copy things successfully speculated till now
>>> */
>>>> + rte_memcpy(to_next, from, last_spec *
>>> sizeof(from[0]));
>>>> + from += last_spec;
>>>> + to_next += last_spec;
>>>> + held += last_spec;
>>>> + last_spec = 0;
>>>> +
>>>> + /* Next0 */
>>>> + if (next_index == next0) {
>>>> + to_next[0] = from[0];
>>>> + to_next++;
>>>> + held++;
>>>> + } else {
>>>> + rte_node_enqueue_x1(graph, node,
>>> next0,
>>>> + from[0]);
>>>> + }
>>>> +
>>>> + /* Next1 */
>>>> + if (next_index == next1) {
>>>> + to_next[0] = from[1];
>>>> + to_next++;
>>>> + held++;
>>>> + } else {
>>>> + rte_node_enqueue_x1(graph, node,
>>> next1,
>>>> + from[1]);
>>>> + }
>>>> +
>>>> + /* Next2 */
>>>> + if (next_index == next2) {
>>>> + to_next[0] = from[2];
>>>> + to_next++;
>>>> + held++;
>>>> + } else {
>>>> + rte_node_enqueue_x1(graph, node,
>>> next2,
>>>> + from[2]);
>>>> + }
>>>> +
>>>> + /* Next3 */
>>>> + if (next_index == next3) {
>>>> + to_next[0] = from[3];
>>>> + to_next++;
>>>> + held++;
>>>> + } else {
>>>> + rte_node_enqueue_x1(graph, node,
>>> next3,
>>>> + from[3]);
>>>> + }
>>>> +
>>>> + from += 4;
>>>> +
>>>> + } else {
>>>> + last_spec += 4;
>>>> + }
>>>> + }
>>>> +
>>>> + while (n_left_from > 0) {
>>>> + uint32_t next_hop;
>>>> +
>>>> + mbuf0 = pkts[0];
>>>> +
>>>> + pkts += 1;
>>>> + n_left_from -= 1;
>>>> +
>>>> + /* Extract DIP of mbuf0 */
>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>> rte_ether_hdr *);
>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
>>>> hdr_checksum;
>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>> time_to_live;
>>>> +
>>>> + rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr-
>>>> dst_addr),
>>>> + &next_hop);
>>>> + next_hop = (rc == 0) ? next_hop : drop_nh;
>>>> +
>>>> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop &
>>> 0xFFFF;
>>>> + next0 = (next_hop >> 16);
>>>> +
>>>> + if (unlikely(next_index ^ next0)) {
>>>> + /* Copy things successfully speculated till now
>>> */
>>>> + rte_memcpy(to_next, from, last_spec *
>>> sizeof(from[0]));
>>>> + from += last_spec;
>>>> + to_next += last_spec;
>>>> + held += last_spec;
>>>> + last_spec = 0;
>>>> +
>>>> + rte_node_enqueue_x1(graph, node, next0,
>>> from[0]);
>>>> + from += 1;
>>>> + } else {
>>>> + last_spec += 1;
>>>> + }
>>>> + }
>>>> +
>>>> + /* !!! Home run !!! */
>>>> + if (likely(last_spec == nb_objs)) {
>>>> + rte_node_next_stream_move(graph, node,
>>> next_index);
>>>> + return nb_objs;
>>>> + }
>>>> +
>>>> + held += last_spec;
>>>> + /* Copy things successfully speculated till now */
>>>> + rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
>>>> + rte_node_next_stream_put(graph, node, next_index, held);
>>>> +
>>>> + return nb_objs;
>>>> +}
>>>> +
>>>> #else
>>>>
>>>> static uint16_t
>>>>
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-19 16:13 ` Pavan Nikhilesh Bhagavatula
@ 2020-03-20 9:14 ` Ray Kinsella
2020-03-24 9:40 ` Pavan Nikhilesh Bhagavatula
0 siblings, 1 reply; 219+ messages in thread
From: Ray Kinsella @ 2020-03-20 9:14 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula, Jerin Jacob Kollanukkaran,
Nithin Kumar Dabilpuram
Cc: dev, thomas, david.marchand, mattias.ronnblom, Kiran Kumar Kokkilagadda
On 19/03/2020 16:13, Pavan Nikhilesh Bhagavatula wrote:
>
>
>> -----Original Message-----
>> From: Ray Kinsella <mdr@ashroe.eu>
>> Sent: Thursday, March 19, 2020 9:21 PM
>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Jerin
>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar Dabilpuram
>> <ndabilpuram@marvell.com>
>> Cc: dev@dpdk.org; thomas@monjalon.net;
>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com; Kiran
>> Kumar Kokkilagadda <kirankumark@marvell.com>
>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup
>> for x86
>>
>>
>>
>> On 19/03/2020 14:22, Pavan Nikhilesh Bhagavatula wrote:
>>>> On 18/03/2020 21:35, jerinj@marvell.com wrote:
>>>>> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>
>>>>> Add IPv4 lookup process function for ip4_lookup
>>>>> rte_node. This node performs LPM lookup using x86_64
>>>>> vector supported RTE_LPM API on every packet received
>>>>> and forwards it to a next node that is identified by
>>>>> lookup result.
>>>>>
>>>>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
>>>>> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
>>>>> ---
>>>>> lib/librte_node/ip4_lookup.c | 245
>>>> +++++++++++++++++++++++++++++++++++
>>>>> 1 file changed, 245 insertions(+)
>>>>>
>>>>> diff --git a/lib/librte_node/ip4_lookup.c
>>>> b/lib/librte_node/ip4_lookup.c
>>>>> index d7fcd1158..c003e9c91 100644
>>>>> --- a/lib/librte_node/ip4_lookup.c
>>>>> +++ b/lib/librte_node/ip4_lookup.c
>>>>> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct
>> rte_graph
>>>> *graph, struct rte_node *node,
>>>>> return nb_objs;
>>>>> }
>>>>>
>>>>> +#elif defined(RTE_ARCH_X86)
>>>>> +
>>>>> +/* X86 SSE */
>>>>> +static uint16_t
>>>>> +ip4_lookup_node_process(struct rte_graph *graph, struct
>> rte_node
>>>> *node,
>>>>> + void **objs, uint16_t nb_objs)
>>>>> +{
>>>>> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
>>>>> + rte_edge_t next0, next1, next2, next3, next_index;
>>>>> + struct rte_ipv4_hdr *ipv4_hdr;
>>>>> + struct rte_ether_hdr *eth_hdr;
>>>>> + uint32_t ip0, ip1, ip2, ip3;
>>>>> + void **to_next, **from;
>>>>> + uint16_t last_spec = 0;
>>>>> + uint16_t n_left_from;
>>>>> + struct rte_lpm *lpm;
>>>>> + uint16_t held = 0;
>>>>> + uint32_t drop_nh;
>>>>> + rte_xmm_t dst;
>>>>> + __m128i dip; /* SSE register */
>>>>> + int rc, i;
>>>>> +
>>>>> + /* Speculative next */
>>>>> + next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
>>>>> + /* Drop node */
>>>>> + drop_nh =
>>>> ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
>>>>> +
>>>>> + /* Get socket specific LPM from ctx */
>>>>> + lpm = *((struct rte_lpm **)node->ctx);
>>>>> +
>>>>> + pkts = (struct rte_mbuf **)objs;
>>>>> + from = objs;
>>>>> + n_left_from = nb_objs;
>>>>
>>>> I doubt this initial prefetch of the first 4 packets has any benefit.
>>>
>>> Ack will remove in v2 for x86.
>>>
>>>>
>>>>> + if (n_left_from >= 4) {
>>>>> + for (i = 0; i < 4; i++) {
>>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
>>>>> + struct rte_ether_hdr
>>>> *) +
>>>>> + 1);
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + /* Get stream for the speculated next node */
>>>>> + to_next = rte_node_next_stream_get(graph, node,
>>>> next_index, nb_objs);
>>>>
>>>> Suggest you don't reuse the hand-unrolling optimization from FD.io
>>>> VPP.
>>>> I have never found any performance benefit from them, and they
>>>> make the code unnecessarily verbose.
>>>>
>>>
>>> How would be take the benefit of rte_lpm_lookupx4 without
>> unrolling the loop?.
>>> Also, in future if we are using rte_rib and fib with a CPU supporting
>> wider SIMD we might
>>> need to unroll them further (AVX256 AND 512 currently
>> rte_lpm_lookup uses only 128bit
>>> since it is only uses SSE extension).
>>
>> Let the compiler do it for you, but using a constant vector length.
>> for (int i=0; i < 4; ++i) { ... }
>>
>
> Ok, I think I misunderstood the previous comment.
> It was only for the prefetches in the loop right?
no, it was for all the needless repetition.
hand-unrolling loops serve no purpose but to add verbosity.
>
>>>
>>>>
>>>>> + while (n_left_from >= 4) {
>>>>> + /* Prefetch next-next mbufs */
>>>>> + if (likely(n_left_from >= 11)) {
>>>>> + rte_prefetch0(pkts[8]);
>>>>> + rte_prefetch0(pkts[9]);
>>>>> + rte_prefetch0(pkts[10]);
>>>>> + rte_prefetch0(pkts[11]);
>>>>> + }
>>>>> +
>>>>> + /* Prefetch next mbuf data */
>>>>> + if (likely(n_left_from >= 7)) {
>>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
>>>>> + struct rte_ether_hdr
>>>> *) +
>>>>> + 1);
>>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
>>>>> + struct rte_ether_hdr
>>>> *) +
>>>>> + 1);
>>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
>>>>> + struct rte_ether_hdr
>>>> *) +
>>>>> + 1);
>>>>> + rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
>>>>> + struct rte_ether_hdr
>>>> *) +
>>>>> + 1);
>>>>> + }
>>>>> +
>>>>> + mbuf0 = pkts[0];
>>>>> + mbuf1 = pkts[1];
>>>>> + mbuf2 = pkts[2];
>>>>> + mbuf3 = pkts[3];
>>>>> +
>>>>> + pkts += 4;
>>>>> + n_left_from -= 4;
>>>>> +
>>>>> + /* Extract DIP of mbuf0 */
>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>> rte_ether_hdr *);
>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>> + ip0 = ipv4_hdr->dst_addr;
>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
>>>>> hdr_checksum;
>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>> time_to_live;
>>>>> +
>>>>> + /* Extract DIP of mbuf1 */
>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct
>>>> rte_ether_hdr *);
>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>> + ip1 = ipv4_hdr->dst_addr;
>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>> + rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr-
>>>>> hdr_checksum;
>>>>> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-
>>>>> time_to_live;
>>>>> +
>>>>> + /* Extract DIP of mbuf2 */
>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct
>>>> rte_ether_hdr *);
>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>> + ip2 = ipv4_hdr->dst_addr;
>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>> + rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr-
>>>>> hdr_checksum;
>>>>> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-
>>>>> time_to_live;
>>>>> +
>>>>> + /* Extract DIP of mbuf3 */
>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct
>>>> rte_ether_hdr *);
>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>> + ip3 = ipv4_hdr->dst_addr;
>>>>> +
>>>>> + /* Prepare for lookup x4 */
>>>>> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
>>>>> +
>>>>> + /* Byte swap 4 IPV4 addresses. */
>>>>> + const __m128i bswap_mask = _mm_set_epi8(
>>>>> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
>>>>> + dip = _mm_shuffle_epi8(dip, bswap_mask);
>>>>> +
>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>> + rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr-
>>>>> hdr_checksum;
>>>>> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-
>>>>> time_to_live;
>>>>> +
>>>>> + /* Perform LPM lookup to get NH and next node */
>>>>> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
>>>>> +
>>>>> + /* Extract next node id and NH */
>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] &
>>>> 0xFFFF;
>>>>> + next0 = (dst.u32[0] >> 16);
>>>>> +
>>>>> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] &
>>>> 0xFFFF;
>>>>> + next1 = (dst.u32[1] >> 16);
>>>>> +
>>>>> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] &
>>>> 0xFFFF;
>>>>> + next2 = (dst.u32[2] >> 16);
>>>>> +
>>>>> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] &
>>>> 0xFFFF;
>>>>> + next3 = (dst.u32[3] >> 16);
>>>>> +
>>>>> + /* Enqueue four to next node */
>>>>> + rte_edge_t fix_spec =
>>>>> + (next_index ^ next0) | (next_index ^ next1) |
>>>>> + (next_index ^ next2) | (next_index ^ next3);
>>>>> +
>>>>> + if (unlikely(fix_spec)) {
>>>>> + /* Copy things successfully speculated till now
>>>> */
>>>>> + rte_memcpy(to_next, from, last_spec *
>>>> sizeof(from[0]));
>>>>> + from += last_spec;
>>>>> + to_next += last_spec;
>>>>> + held += last_spec;
>>>>> + last_spec = 0;
>>>>> +
>>>>> + /* Next0 */
>>>>> + if (next_index == next0) {
>>>>> + to_next[0] = from[0];
>>>>> + to_next++;
>>>>> + held++;
>>>>> + } else {
>>>>> + rte_node_enqueue_x1(graph, node,
>>>> next0,
>>>>> + from[0]);
>>>>> + }
>>>>> +
>>>>> + /* Next1 */
>>>>> + if (next_index == next1) {
>>>>> + to_next[0] = from[1];
>>>>> + to_next++;
>>>>> + held++;
>>>>> + } else {
>>>>> + rte_node_enqueue_x1(graph, node,
>>>> next1,
>>>>> + from[1]);
>>>>> + }
>>>>> +
>>>>> + /* Next2 */
>>>>> + if (next_index == next2) {
>>>>> + to_next[0] = from[2];
>>>>> + to_next++;
>>>>> + held++;
>>>>> + } else {
>>>>> + rte_node_enqueue_x1(graph, node,
>>>> next2,
>>>>> + from[2]);
>>>>> + }
>>>>> +
>>>>> + /* Next3 */
>>>>> + if (next_index == next3) {
>>>>> + to_next[0] = from[3];
>>>>> + to_next++;
>>>>> + held++;
>>>>> + } else {
>>>>> + rte_node_enqueue_x1(graph, node,
>>>> next3,
>>>>> + from[3]);
>>>>> + }
>>>>> +
>>>>> + from += 4;
>>>>> +
>>>>> + } else {
>>>>> + last_spec += 4;
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + while (n_left_from > 0) {
>>>>> + uint32_t next_hop;
>>>>> +
>>>>> + mbuf0 = pkts[0];
>>>>> +
>>>>> + pkts += 1;
>>>>> + n_left_from -= 1;
>>>>> +
>>>>> + /* Extract DIP of mbuf0 */
>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>> rte_ether_hdr *);
>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr-
>>>>> hdr_checksum;
>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>> time_to_live;
>>>>> +
>>>>> + rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr-
>>>>> dst_addr),
>>>>> + &next_hop);
>>>>> + next_hop = (rc == 0) ? next_hop : drop_nh;
>>>>> +
>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop &
>>>> 0xFFFF;
>>>>> + next0 = (next_hop >> 16);
>>>>> +
>>>>> + if (unlikely(next_index ^ next0)) {
>>>>> + /* Copy things successfully speculated till now
>>>> */
>>>>> + rte_memcpy(to_next, from, last_spec *
>>>> sizeof(from[0]));
>>>>> + from += last_spec;
>>>>> + to_next += last_spec;
>>>>> + held += last_spec;
>>>>> + last_spec = 0;
>>>>> +
>>>>> + rte_node_enqueue_x1(graph, node, next0,
>>>> from[0]);
>>>>> + from += 1;
>>>>> + } else {
>>>>> + last_spec += 1;
>>>>> + }
>>>>> + }
>>>>> +
>>>>> + /* !!! Home run !!! */
>>>>> + if (likely(last_spec == nb_objs)) {
>>>>> + rte_node_next_stream_move(graph, node,
>>>> next_index);
>>>>> + return nb_objs;
>>>>> + }
>>>>> +
>>>>> + held += last_spec;
>>>>> + /* Copy things successfully speculated till now */
>>>>> + rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
>>>>> + rte_node_next_stream_put(graph, node, next_index, held);
>>>>> +
>>>>> + return nb_objs;
>>>>> +}
>>>>> +
>>>>> #else
>>>>>
>>>>> static uint16_t
>>>>>
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-20 9:14 ` Ray Kinsella
@ 2020-03-24 9:40 ` Pavan Nikhilesh Bhagavatula
2020-03-24 14:38 ` Ray Kinsella
0 siblings, 1 reply; 219+ messages in thread
From: Pavan Nikhilesh Bhagavatula @ 2020-03-24 9:40 UTC (permalink / raw)
To: Ray Kinsella, Jerin Jacob Kollanukkaran, Nithin Kumar Dabilpuram
Cc: dev, thomas, david.marchand, mattias.ronnblom, Kiran Kumar Kokkilagadda
Hi Ray,
I have tried to avoid hand unrolling loops and found the following observations.
1. Although it decreases LOC it also takes away readability too.
Example:
Avoiding unrolled code below
priv[0].u64[0] = rte_node_mbuf_priv1(mbuf[0])->u;
priv[0].u64[1] = rte_node_mbuf_priv1(mbuf[1])->u;
priv[1].u64[0] = rte_node_mbuf_priv1(mbuf[2])->u;
priv[1].u64[1] = rte_node_mbuf_priv1(mbuf[3])->u;
/* Increment checksum by one. */
priv[0].u32[1] += rte_cpu_to_be_16(0x0100);
priv[0].u32[3] += rte_cpu_to_be_16(0x0100);
priv[1].u32[1] += rte_cpu_to_be_16(0x0100);
priv[1].u32[3] += rte_cpu_to_be_16(0x0100);
d = rte_pktmbuf_mtod(mbuf[0], void *);
rte_memcpy(d, nh[priv[0].u16[0]].rewrite_data,
nh[priv[0].u16[0]].rewrite_len);
next[0] = nh[priv[0].u16[0]].tx_node;
ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
sizeof(struct rte_ether_hdr));
ip->time_to_live = priv[0].u16[1] - 1;
ip->hdr_checksum = priv[0].u16[2] + priv[0].u16[3];
d = rte_pktmbuf_mtod(mbuf[1], void *);
rte_memcpy(d, nh[priv[0].u16[4]].rewrite_data,
nh[priv[0].u16[4]].rewrite_len);
next[1] = nh[priv[0].u16[4]].tx_node;
ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
sizeof(struct rte_ether_hdr));
ip->time_to_live = priv[0].u16[5] - 1;
ip->hdr_checksum = priv[0].u16[6] + priv[0].u16[7];
d = rte_pktmbuf_mtod(mbuf[2], void *);
rte_memcpy(d, nh[priv[1].u16[0]].rewrite_data,
nh[priv[1].u16[0]].rewrite_len);
next[2] = nh[priv[1].u16[0]].tx_node;
ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
sizeof(struct rte_ether_hdr));
ip->time_to_live = priv[1].u16[1] - 1;
ip->hdr_checksum = priv[1].u16[2] + priv[1].u16[3];
d = rte_pktmbuf_mtod(mbuf[3], void *);
rte_memcpy(d, nh[priv[1].u16[4]].rewrite_data,
nh[priv[1].u16[4]].rewrite_len);
next[3] = nh[priv[1].u16[4]].tx_node;
ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
sizeof(struct rte_ether_hdr));
ip->time_to_live = priv[1].u16[5] - 1;
ip->hdr_checksum = priv[1].u16[6] + priv[1].u16[7];
Leads to something like:
for (i = 0, j = 0; i < BUF_PER_LOOP; i += 2, j++) {
priv[j].u64[0] = rte_node_mbuf_priv1(mbuf[i])->u;
priv[j].u32[1] += rte_cpu_to_be_16(0x0100);
d = rte_pktmbuf_mtod(mbuf[i], void *);
ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
sizeof(struct rte_ether_hdr));
ip->time_to_live = priv[j].u16[1] - 1;
ip->hdr_checksum = priv[j].u16[2] + priv[j].u16[3];
rte_memcpy(d, nh[priv[j].u16[0]].rewrite_data,
nh[priv[j].u16[0]].rewrite_len);
next[i] = nh[priv[j].u16[0]].tx_node;
priv[j].u64[1] = rte_node_mbuf_priv1(mbuf[i + 1])->u;
priv[j].u32[3] += rte_cpu_to_be_16(0x0100);
d = rte_pktmbuf_mtod(mbuf[i + 1], void *);
ip = (struct rte_ipv4_hdr *)((uint8_t *)d +
sizeof(struct rte_ether_hdr));
ip->time_to_live = priv[j].u16[5] - 1;
ip->hdr_checksum = priv[j].u16[6] + priv[j].u16[7];
rte_memcpy(d, nh[priv[j].u16[4]].rewrite_data,
nh[priv[j].u16[4]].rewrite_len);
next[i + 1] = nh[priv[j].u16[4]].tx_node;
}
Which is kind of unreadable.
2. Not all compilers are made equal. I found that most of the compilers don’t
Unroll the loop above even when compiled with `-funroll-all-loops`.
I have checked with following compilers:
GCC 9.2.0
Clang 9.0.1
Aarch64 GCC 7.3.0
Aarch64 GCC 9.2.0
3. Performance wise I see a lot of degradation on our platform at least 13%.
On IA with a Broadwell(Xeon E5-2690) and i40e the performance remain same w.r.t Rx/Tx since the
hotspot is in the Tx path of the driver which limits the per core capability.
But the performance difference in number of cycles per node can be seen below:
Hand unrolling:
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|Node |calls |objs |realloc_count |objs/call |objs/sec(10E6) |cycles/call|
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|ip4_lookup |7765918 |248509344 |1 |32.000 |27.725408 |779.0000 |
|ip4_rewrite |7765925 |248509568 |1 |32.000 |27.725408 |425.0000 |
|ethdev_tx-1 |7765927 |204056223 |1 |26.000 |22.762720 |597.0000 |
|pkt_drop |1389170 |44453409 |1 |32.000 |4.962688 |298.0000 |
|ethdev_rx-0-0 |63604111 |248509792 |2 |32.000 |27.725408 |982.0000 |
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
W/o unrolling:
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|Node |calls |objs |realloc_count |objs/call |objs/sec(10E6) |cycles/call|
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
|ip4_lookup |18864640 |603668448 |1 |32.000 |26.051328 |828.0000 |
|ip4_rewrite |18864646 |603668640 |1 |32.000 |26.051328 |534.0000 |
|ethdev_tx-1 |18864648 |527874175 |1 |27.000 |22.780256 |633.0000 |
|pkt_drop |2368580 |75794529 |1 |32.000 |3.271072 |286.0000 |
|ethdev_rx-0-0 |282058226 |603668864 |2 |32.000 |26.051328 |994.0000 |
+-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
Considering the above findings we would like to continue unrolling the loops by hand.
Regards,
Pavan.
>-----Original Message-----
>From: Ray Kinsella <mdr@ashroe.eu>
>Sent: Friday, March 20, 2020 2:44 PM
>To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Jerin
>Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar Dabilpuram
><ndabilpuram@marvell.com>
>Cc: dev@dpdk.org; thomas@monjalon.net;
>david.marchand@redhat.com; mattias.ronnblom@ericsson.com; Kiran
>Kumar Kokkilagadda <kirankumark@marvell.com>
>Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup
>for x86
>
>
>
>On 19/03/2020 16:13, Pavan Nikhilesh Bhagavatula wrote:
>>
>>
>>> -----Original Message-----
>>> From: Ray Kinsella <mdr@ashroe.eu>
>>> Sent: Thursday, March 19, 2020 9:21 PM
>>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>Jerin
>>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar
>Dabilpuram
>>> <ndabilpuram@marvell.com>
>>> Cc: dev@dpdk.org; thomas@monjalon.net;
>>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com;
>Kiran
>>> Kumar Kokkilagadda <kirankumark@marvell.com>
>>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4
>lookup
>>> for x86
>>>
>>>
>>>
>>> On 19/03/2020 14:22, Pavan Nikhilesh Bhagavatula wrote:
>>>>> On 18/03/2020 21:35, jerinj@marvell.com wrote:
>>>>>> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>>
>>>>>> Add IPv4 lookup process function for ip4_lookup
>>>>>> rte_node. This node performs LPM lookup using x86_64
>>>>>> vector supported RTE_LPM API on every packet received
>>>>>> and forwards it to a next node that is identified by
>>>>>> lookup result.
>>>>>>
>>>>>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
>>>>>> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
>>>>>> ---
>>>>>> lib/librte_node/ip4_lookup.c | 245
>>>>> +++++++++++++++++++++++++++++++++++
>>>>>> 1 file changed, 245 insertions(+)
>>>>>>
>>>>>> diff --git a/lib/librte_node/ip4_lookup.c
>>>>> b/lib/librte_node/ip4_lookup.c
>>>>>> index d7fcd1158..c003e9c91 100644
>>>>>> --- a/lib/librte_node/ip4_lookup.c
>>>>>> +++ b/lib/librte_node/ip4_lookup.c
>>>>>> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct
>>> rte_graph
>>>>> *graph, struct rte_node *node,
>>>>>> return nb_objs;
>>>>>> }
>>>>>>
>>>>>> +#elif defined(RTE_ARCH_X86)
>>>>>> +
>>>>>> +/* X86 SSE */
>>>>>> +static uint16_t
>>>>>> +ip4_lookup_node_process(struct rte_graph *graph, struct
>>> rte_node
>>>>> *node,
>>>>>> + void **objs, uint16_t nb_objs)
>>>>>> +{
>>>>>> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3,
>**pkts;
>>>>>> + rte_edge_t next0, next1, next2, next3, next_index;
>>>>>> + struct rte_ipv4_hdr *ipv4_hdr;
>>>>>> + struct rte_ether_hdr *eth_hdr;
>>>>>> + uint32_t ip0, ip1, ip2, ip3;
>>>>>> + void **to_next, **from;
>>>>>> + uint16_t last_spec = 0;
>>>>>> + uint16_t n_left_from;
>>>>>> + struct rte_lpm *lpm;
>>>>>> + uint16_t held = 0;
>>>>>> + uint32_t drop_nh;
>>>>>> + rte_xmm_t dst;
>>>>>> + __m128i dip; /* SSE register */
>>>>>> + int rc, i;
>>>>>> +
>>>>>> + /* Speculative next */
>>>>>> + next_index =
>RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
>>>>>> + /* Drop node */
>>>>>> + drop_nh =
>>>>> ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
>>>>>> +
>>>>>> + /* Get socket specific LPM from ctx */
>>>>>> + lpm = *((struct rte_lpm **)node->ctx);
>>>>>> +
>>>>>> + pkts = (struct rte_mbuf **)objs;
>>>>>> + from = objs;
>>>>>> + n_left_from = nb_objs;
>>>>>
>>>>> I doubt this initial prefetch of the first 4 packets has any benefit.
>>>>
>>>> Ack will remove in v2 for x86.
>>>>
>>>>>
>>>>>> + if (n_left_from >= 4) {
>>>>>> + for (i = 0; i < 4; i++) {
>>>>>> +
> rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
>>>>>> + struct
>rte_ether_hdr
>>>>> *) +
>>>>>> + 1);
>>>>>> + }
>>>>>> + }
>>>>>> +
>>>>>> + /* Get stream for the speculated next node */
>>>>>> + to_next = rte_node_next_stream_get(graph, node,
>>>>> next_index, nb_objs);
>>>>>
>>>>> Suggest you don't reuse the hand-unrolling optimization from
>FD.io
>>>>> VPP.
>>>>> I have never found any performance benefit from them, and they
>>>>> make the code unnecessarily verbose.
>>>>>
>>>>
>>>> How would be take the benefit of rte_lpm_lookupx4 without
>>> unrolling the loop?.
>>>> Also, in future if we are using rte_rib and fib with a CPU supporting
>>> wider SIMD we might
>>>> need to unroll them further (AVX256 AND 512 currently
>>> rte_lpm_lookup uses only 128bit
>>>> since it is only uses SSE extension).
>>>
>>> Let the compiler do it for you, but using a constant vector length.
>>> for (int i=0; i < 4; ++i) { ... }
>>>
>>
>> Ok, I think I misunderstood the previous comment.
>> It was only for the prefetches in the loop right?
>
>
>no, it was for all the needless repetition.
>hand-unrolling loops serve no purpose but to add verbosity.
>
>>
>>>>
>>>>>
>>>>>> + while (n_left_from >= 4) {
>>>>>> + /* Prefetch next-next mbufs */
>>>>>> + if (likely(n_left_from >= 11)) {
>>>>>> + rte_prefetch0(pkts[8]);
>>>>>> + rte_prefetch0(pkts[9]);
>>>>>> + rte_prefetch0(pkts[10]);
>>>>>> + rte_prefetch0(pkts[11]);
>>>>>> + }
>>>>>> +
>>>>>> + /* Prefetch next mbuf data */
>>>>>> + if (likely(n_left_from >= 7)) {
>>>>>> +
> rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
>>>>>> + struct
>rte_ether_hdr
>>>>> *) +
>>>>>> + 1);
>>>>>> +
> rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
>>>>>> + struct
>rte_ether_hdr
>>>>> *) +
>>>>>> + 1);
>>>>>> +
> rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
>>>>>> + struct
>rte_ether_hdr
>>>>> *) +
>>>>>> + 1);
>>>>>> +
> rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
>>>>>> + struct
>rte_ether_hdr
>>>>> *) +
>>>>>> + 1);
>>>>>> + }
>>>>>> +
>>>>>> + mbuf0 = pkts[0];
>>>>>> + mbuf1 = pkts[1];
>>>>>> + mbuf2 = pkts[2];
>>>>>> + mbuf3 = pkts[3];
>>>>>> +
>>>>>> + pkts += 4;
>>>>>> + n_left_from -= 4;
>>>>>> +
>>>>>> + /* Extract DIP of mbuf0 */
>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>>> rte_ether_hdr *);
>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>> + ip0 = ipv4_hdr->dst_addr;
>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum =
>ipv4_hdr-
>>>>>> hdr_checksum;
>>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>>> time_to_live;
>>>>>> +
>>>>>> + /* Extract DIP of mbuf1 */
>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct
>>>>> rte_ether_hdr *);
>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>> + ip1 = ipv4_hdr->dst_addr;
>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>> + rte_node_mbuf_priv1(mbuf1)->cksum =
>ipv4_hdr-
>>>>>> hdr_checksum;
>>>>>> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-
>>>>>> time_to_live;
>>>>>> +
>>>>>> + /* Extract DIP of mbuf2 */
>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct
>>>>> rte_ether_hdr *);
>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>> + ip2 = ipv4_hdr->dst_addr;
>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>> + rte_node_mbuf_priv1(mbuf2)->cksum =
>ipv4_hdr-
>>>>>> hdr_checksum;
>>>>>> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-
>>>>>> time_to_live;
>>>>>> +
>>>>>> + /* Extract DIP of mbuf3 */
>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct
>>>>> rte_ether_hdr *);
>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>> + ip3 = ipv4_hdr->dst_addr;
>>>>>> +
>>>>>> + /* Prepare for lookup x4 */
>>>>>> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
>>>>>> +
>>>>>> + /* Byte swap 4 IPV4 addresses. */
>>>>>> + const __m128i bswap_mask = _mm_set_epi8(
>>>>>> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1,
>2, 3);
>>>>>> + dip = _mm_shuffle_epi8(dip, bswap_mask);
>>>>>> +
>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>> + rte_node_mbuf_priv1(mbuf3)->cksum =
>ipv4_hdr-
>>>>>> hdr_checksum;
>>>>>> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-
>>>>>> time_to_live;
>>>>>> +
>>>>>> + /* Perform LPM lookup to get NH and next
>node */
>>>>>> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
>>>>>> +
>>>>>> + /* Extract next node id and NH */
>>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0]
>&
>>>>> 0xFFFF;
>>>>>> + next0 = (dst.u32[0] >> 16);
>>>>>> +
>>>>>> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1]
>&
>>>>> 0xFFFF;
>>>>>> + next1 = (dst.u32[1] >> 16);
>>>>>> +
>>>>>> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2]
>&
>>>>> 0xFFFF;
>>>>>> + next2 = (dst.u32[2] >> 16);
>>>>>> +
>>>>>> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3]
>&
>>>>> 0xFFFF;
>>>>>> + next3 = (dst.u32[3] >> 16);
>>>>>> +
>>>>>> + /* Enqueue four to next node */
>>>>>> + rte_edge_t fix_spec =
>>>>>> + (next_index ^ next0) | (next_index ^
>next1) |
>>>>>> + (next_index ^ next2) | (next_index ^
>next3);
>>>>>> +
>>>>>> + if (unlikely(fix_spec)) {
>>>>>> + /* Copy things successfully speculated
>till now
>>>>> */
>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>>> sizeof(from[0]));
>>>>>> + from += last_spec;
>>>>>> + to_next += last_spec;
>>>>>> + held += last_spec;
>>>>>> + last_spec = 0;
>>>>>> +
>>>>>> + /* Next0 */
>>>>>> + if (next_index == next0) {
>>>>>> + to_next[0] = from[0];
>>>>>> + to_next++;
>>>>>> + held++;
>>>>>> + } else {
>>>>>> + rte_node_enqueue_x1(graph,
>node,
>>>>> next0,
>>>>>> + from[0]);
>>>>>> + }
>>>>>> +
>>>>>> + /* Next1 */
>>>>>> + if (next_index == next1) {
>>>>>> + to_next[0] = from[1];
>>>>>> + to_next++;
>>>>>> + held++;
>>>>>> + } else {
>>>>>> + rte_node_enqueue_x1(graph,
>node,
>>>>> next1,
>>>>>> + from[1]);
>>>>>> + }
>>>>>> +
>>>>>> + /* Next2 */
>>>>>> + if (next_index == next2) {
>>>>>> + to_next[0] = from[2];
>>>>>> + to_next++;
>>>>>> + held++;
>>>>>> + } else {
>>>>>> + rte_node_enqueue_x1(graph,
>node,
>>>>> next2,
>>>>>> + from[2]);
>>>>>> + }
>>>>>> +
>>>>>> + /* Next3 */
>>>>>> + if (next_index == next3) {
>>>>>> + to_next[0] = from[3];
>>>>>> + to_next++;
>>>>>> + held++;
>>>>>> + } else {
>>>>>> + rte_node_enqueue_x1(graph,
>node,
>>>>> next3,
>>>>>> + from[3]);
>>>>>> + }
>>>>>> +
>>>>>> + from += 4;
>>>>>> +
>>>>>> + } else {
>>>>>> + last_spec += 4;
>>>>>> + }
>>>>>> + }
>>>>>> +
>>>>>> + while (n_left_from > 0) {
>>>>>> + uint32_t next_hop;
>>>>>> +
>>>>>> + mbuf0 = pkts[0];
>>>>>> +
>>>>>> + pkts += 1;
>>>>>> + n_left_from -= 1;
>>>>>> +
>>>>>> + /* Extract DIP of mbuf0 */
>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>>> rte_ether_hdr *);
>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum =
>ipv4_hdr-
>>>>>> hdr_checksum;
>>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>>> time_to_live;
>>>>>> +
>>>>>> + rc = rte_lpm_lookup(lpm,
>rte_be_to_cpu_32(ipv4_hdr-
>>>>>> dst_addr),
>>>>>> + &next_hop);
>>>>>> + next_hop = (rc == 0) ? next_hop : drop_nh;
>>>>>> +
>>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop
>&
>>>>> 0xFFFF;
>>>>>> + next0 = (next_hop >> 16);
>>>>>> +
>>>>>> + if (unlikely(next_index ^ next0)) {
>>>>>> + /* Copy things successfully speculated
>till now
>>>>> */
>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>>> sizeof(from[0]));
>>>>>> + from += last_spec;
>>>>>> + to_next += last_spec;
>>>>>> + held += last_spec;
>>>>>> + last_spec = 0;
>>>>>> +
>>>>>> + rte_node_enqueue_x1(graph, node,
>next0,
>>>>> from[0]);
>>>>>> + from += 1;
>>>>>> + } else {
>>>>>> + last_spec += 1;
>>>>>> + }
>>>>>> + }
>>>>>> +
>>>>>> + /* !!! Home run !!! */
>>>>>> + if (likely(last_spec == nb_objs)) {
>>>>>> + rte_node_next_stream_move(graph, node,
>>>>> next_index);
>>>>>> + return nb_objs;
>>>>>> + }
>>>>>> +
>>>>>> + held += last_spec;
>>>>>> + /* Copy things successfully speculated till now */
>>>>>> + rte_memcpy(to_next, from, last_spec *
>sizeof(from[0]));
>>>>>> + rte_node_next_stream_put(graph, node, next_index,
>held);
>>>>>> +
>>>>>> + return nb_objs;
>>>>>> +}
>>>>>> +
>>>>>> #else
>>>>>>
>>>>>> static uint16_t
>>>>>>
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-24 9:40 ` Pavan Nikhilesh Bhagavatula
@ 2020-03-24 14:38 ` Ray Kinsella
2020-03-26 9:56 ` Pavan Nikhilesh Bhagavatula
0 siblings, 1 reply; 219+ messages in thread
From: Ray Kinsella @ 2020-03-24 14:38 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula, Jerin Jacob Kollanukkaran,
Nithin Kumar Dabilpuram
Cc: dev, thomas, david.marchand, mattias.ronnblom, Kiran Kumar Kokkilagadda
On 24/03/2020 09:40, Pavan Nikhilesh Bhagavatula wrote:
> Hi Ray,
>
> I have tried to avoid hand unrolling loops and found the following observations.
>
> 1. Although it decreases LOC it also takes away readability too.
> Example:
> Avoiding unrolled code below
[SNIP]
> Which is kind of unreadable.
I am confused - isn't it exactly the same code?
You still haven't completely unrolled the loop either?
I don't know how one is readable and the other is not.
>
> 2. Not all compilers are made equal. I found that most of the compilers don’t
> Unroll the loop above even when compiled with `-funroll-all-loops`.
> I have checked with following compilers:
> GCC 9.2.0
> Clang 9.0.1
> Aarch64 GCC 7.3.0
> Aarch64 GCC 9.2.0
Compilers have been unrolling fixed length loops for as long time - this isn't new technology.
If the compiler isn't unrolling you are doing something that makes it think it is a bad idea.
Hand unrolling the loop isn't the solution, understanding what the compiler is doing is a better idea.
In front of your for loop insert, to indicate to the compiler what you want to do.
#pragma unroll BUF_PER_LOOP
With clang you can ask it why it is not unrolling the loop with the following switches.
(output is verbose, but the reason is in there).
-Rpass=loop-unroll -Rpass-missed=loop-unroll
>
> 3. Performance wise I see a lot of degradation on our platform at least 13%.
Is the loop being unrolled?
> On IA with a Broadwell(Xeon E5-2690) and i40e the performance remain same w.r.t Rx/Tx since the
> hotspot is in the Tx path of the driver which limits the per core capability.
> But the performance difference in number of cycles per node can be seen below:
>
> Hand unrolling:
> +-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
> |Node |calls |objs |realloc_count |objs/call |objs/sec(10E6) |cycles/call|
> +-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
> |ip4_lookup |7765918 |248509344 |1 |32.000 |27.725408 |779.0000 |
> |ip4_rewrite |7765925 |248509568 |1 |32.000 |27.725408 |425.0000 |
> |ethdev_tx-1 |7765927 |204056223 |1 |26.000 |22.762720 |597.0000 |
> |pkt_drop |1389170 |44453409 |1 |32.000 |4.962688 |298.0000 |
> |ethdev_rx-0-0 |63604111 |248509792 |2 |32.000 |27.725408 |982.0000 |
> +-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
>
> W/o unrolling:
>
> +-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
> |Node |calls |objs |realloc_count |objs/call |objs/sec(10E6) |cycles/call|
> +-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
> |ip4_lookup |18864640 |603668448 |1 |32.000 |26.051328 |828.0000 |
> |ip4_rewrite |18864646 |603668640 |1 |32.000 |26.051328 |534.0000 |
> |ethdev_tx-1 |18864648 |527874175 |1 |27.000 |22.780256 |633.0000 |
> |pkt_drop |2368580 |75794529 |1 |32.000 |3.271072 |286.0000 |
> |ethdev_rx-0-0 |282058226 |603668864 |2 |32.000 |26.051328 |994.0000 |
> +-------------------------------+---------------+---------------+---------------+---------------+---------------+-----------+
>
> Considering the above findings we would like to continue unrolling the loops by hand.
>
> Regards,
> Pavan.
>
>> -----Original Message-----
>> From: Ray Kinsella <mdr@ashroe.eu>
>> Sent: Friday, March 20, 2020 2:44 PM
>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Jerin
>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar Dabilpuram
>> <ndabilpuram@marvell.com>
>> Cc: dev@dpdk.org; thomas@monjalon.net;
>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com; Kiran
>> Kumar Kokkilagadda <kirankumark@marvell.com>
>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup
>> for x86
>>
>>
>>
>> On 19/03/2020 16:13, Pavan Nikhilesh Bhagavatula wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Ray Kinsella <mdr@ashroe.eu>
>>>> Sent: Thursday, March 19, 2020 9:21 PM
>>>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>> Jerin
>>>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar
>> Dabilpuram
>>>> <ndabilpuram@marvell.com>
>>>> Cc: dev@dpdk.org; thomas@monjalon.net;
>>>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com;
>> Kiran
>>>> Kumar Kokkilagadda <kirankumark@marvell.com>
>>>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4
>> lookup
>>>> for x86
>>>>
>>>>
>>>>
>>>> On 19/03/2020 14:22, Pavan Nikhilesh Bhagavatula wrote:
>>>>>> On 18/03/2020 21:35, jerinj@marvell.com wrote:
>>>>>>> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>>>
>>>>>>> Add IPv4 lookup process function for ip4_lookup
>>>>>>> rte_node. This node performs LPM lookup using x86_64
>>>>>>> vector supported RTE_LPM API on every packet received
>>>>>>> and forwards it to a next node that is identified by
>>>>>>> lookup result.
>>>>>>>
>>>>>>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>>> Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
>>>>>>> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
>>>>>>> ---
>>>>>>> lib/librte_node/ip4_lookup.c | 245
>>>>>> +++++++++++++++++++++++++++++++++++
>>>>>>> 1 file changed, 245 insertions(+)
>>>>>>>
>>>>>>> diff --git a/lib/librte_node/ip4_lookup.c
>>>>>> b/lib/librte_node/ip4_lookup.c
>>>>>>> index d7fcd1158..c003e9c91 100644
>>>>>>> --- a/lib/librte_node/ip4_lookup.c
>>>>>>> +++ b/lib/librte_node/ip4_lookup.c
>>>>>>> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct
>>>> rte_graph
>>>>>> *graph, struct rte_node *node,
>>>>>>> return nb_objs;
>>>>>>> }
>>>>>>>
>>>>>>> +#elif defined(RTE_ARCH_X86)
>>>>>>> +
>>>>>>> +/* X86 SSE */
>>>>>>> +static uint16_t
>>>>>>> +ip4_lookup_node_process(struct rte_graph *graph, struct
>>>> rte_node
>>>>>> *node,
>>>>>>> + void **objs, uint16_t nb_objs)
>>>>>>> +{
>>>>>>> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3,
>> **pkts;
>>>>>>> + rte_edge_t next0, next1, next2, next3, next_index;
>>>>>>> + struct rte_ipv4_hdr *ipv4_hdr;
>>>>>>> + struct rte_ether_hdr *eth_hdr;
>>>>>>> + uint32_t ip0, ip1, ip2, ip3;
>>>>>>> + void **to_next, **from;
>>>>>>> + uint16_t last_spec = 0;
>>>>>>> + uint16_t n_left_from;
>>>>>>> + struct rte_lpm *lpm;
>>>>>>> + uint16_t held = 0;
>>>>>>> + uint32_t drop_nh;
>>>>>>> + rte_xmm_t dst;
>>>>>>> + __m128i dip; /* SSE register */
>>>>>>> + int rc, i;
>>>>>>> +
>>>>>>> + /* Speculative next */
>>>>>>> + next_index =
>> RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
>>>>>>> + /* Drop node */
>>>>>>> + drop_nh =
>>>>>> ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
>>>>>>> +
>>>>>>> + /* Get socket specific LPM from ctx */
>>>>>>> + lpm = *((struct rte_lpm **)node->ctx);
>>>>>>> +
>>>>>>> + pkts = (struct rte_mbuf **)objs;
>>>>>>> + from = objs;
>>>>>>> + n_left_from = nb_objs;
>>>>>>
>>>>>> I doubt this initial prefetch of the first 4 packets has any benefit.
>>>>>
>>>>> Ack will remove in v2 for x86.
>>>>>
>>>>>>
>>>>>>> + if (n_left_from >= 4) {
>>>>>>> + for (i = 0; i < 4; i++) {
>>>>>>> +
>> rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
>>>>>>> + struct
>> rte_ether_hdr
>>>>>> *) +
>>>>>>> + 1);
>>>>>>> + }
>>>>>>> + }
>>>>>>> +
>>>>>>> + /* Get stream for the speculated next node */
>>>>>>> + to_next = rte_node_next_stream_get(graph, node,
>>>>>> next_index, nb_objs);
>>>>>>
>>>>>> Suggest you don't reuse the hand-unrolling optimization from
>> FD.io
>>>>>> VPP.
>>>>>> I have never found any performance benefit from them, and they
>>>>>> make the code unnecessarily verbose.
>>>>>>
>>>>>
>>>>> How would be take the benefit of rte_lpm_lookupx4 without
>>>> unrolling the loop?.
>>>>> Also, in future if we are using rte_rib and fib with a CPU supporting
>>>> wider SIMD we might
>>>>> need to unroll them further (AVX256 AND 512 currently
>>>> rte_lpm_lookup uses only 128bit
>>>>> since it is only uses SSE extension).
>>>>
>>>> Let the compiler do it for you, but using a constant vector length.
>>>> for (int i=0; i < 4; ++i) { ... }
>>>>
>>>
>>> Ok, I think I misunderstood the previous comment.
>>> It was only for the prefetches in the loop right?
>>
>>
>> no, it was for all the needless repetition.
>> hand-unrolling loops serve no purpose but to add verbosity.
>>
>>>
>>>>>
>>>>>>
>>>>>>> + while (n_left_from >= 4) {
>>>>>>> + /* Prefetch next-next mbufs */
>>>>>>> + if (likely(n_left_from >= 11)) {
>>>>>>> + rte_prefetch0(pkts[8]);
>>>>>>> + rte_prefetch0(pkts[9]);
>>>>>>> + rte_prefetch0(pkts[10]);
>>>>>>> + rte_prefetch0(pkts[11]);
>>>>>>> + }
>>>>>>> +
>>>>>>> + /* Prefetch next mbuf data */
>>>>>>> + if (likely(n_left_from >= 7)) {
>>>>>>> +
>> rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
>>>>>>> + struct
>> rte_ether_hdr
>>>>>> *) +
>>>>>>> + 1);
>>>>>>> +
>> rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
>>>>>>> + struct
>> rte_ether_hdr
>>>>>> *) +
>>>>>>> + 1);
>>>>>>> +
>> rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
>>>>>>> + struct
>> rte_ether_hdr
>>>>>> *) +
>>>>>>> + 1);
>>>>>>> +
>> rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
>>>>>>> + struct
>> rte_ether_hdr
>>>>>> *) +
>>>>>>> + 1);
>>>>>>> + }
>>>>>>> +
>>>>>>> + mbuf0 = pkts[0];
>>>>>>> + mbuf1 = pkts[1];
>>>>>>> + mbuf2 = pkts[2];
>>>>>>> + mbuf3 = pkts[3];
>>>>>>> +
>>>>>>> + pkts += 4;
>>>>>>> + n_left_from -= 4;
>>>>>>> +
>>>>>>> + /* Extract DIP of mbuf0 */
>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>>>> rte_ether_hdr *);
>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>> + ip0 = ipv4_hdr->dst_addr;
>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum =
>> ipv4_hdr-
>>>>>>> hdr_checksum;
>>>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>>>> time_to_live;
>>>>>>> +
>>>>>>> + /* Extract DIP of mbuf1 */
>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct
>>>>>> rte_ether_hdr *);
>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>> + ip1 = ipv4_hdr->dst_addr;
>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>> + rte_node_mbuf_priv1(mbuf1)->cksum =
>> ipv4_hdr-
>>>>>>> hdr_checksum;
>>>>>>> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-
>>>>>>> time_to_live;
>>>>>>> +
>>>>>>> + /* Extract DIP of mbuf2 */
>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct
>>>>>> rte_ether_hdr *);
>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>> + ip2 = ipv4_hdr->dst_addr;
>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>> + rte_node_mbuf_priv1(mbuf2)->cksum =
>> ipv4_hdr-
>>>>>>> hdr_checksum;
>>>>>>> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-
>>>>>>> time_to_live;
>>>>>>> +
>>>>>>> + /* Extract DIP of mbuf3 */
>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct
>>>>>> rte_ether_hdr *);
>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>> + ip3 = ipv4_hdr->dst_addr;
>>>>>>> +
>>>>>>> + /* Prepare for lookup x4 */
>>>>>>> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
>>>>>>> +
>>>>>>> + /* Byte swap 4 IPV4 addresses. */
>>>>>>> + const __m128i bswap_mask = _mm_set_epi8(
>>>>>>> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1,
>> 2, 3);
>>>>>>> + dip = _mm_shuffle_epi8(dip, bswap_mask);
>>>>>>> +
>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>> + rte_node_mbuf_priv1(mbuf3)->cksum =
>> ipv4_hdr-
>>>>>>> hdr_checksum;
>>>>>>> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-
>>>>>>> time_to_live;
>>>>>>> +
>>>>>>> + /* Perform LPM lookup to get NH and next
>> node */
>>>>>>> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
>>>>>>> +
>>>>>>> + /* Extract next node id and NH */
>>>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0]
>> &
>>>>>> 0xFFFF;
>>>>>>> + next0 = (dst.u32[0] >> 16);
>>>>>>> +
>>>>>>> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1]
>> &
>>>>>> 0xFFFF;
>>>>>>> + next1 = (dst.u32[1] >> 16);
>>>>>>> +
>>>>>>> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2]
>> &
>>>>>> 0xFFFF;
>>>>>>> + next2 = (dst.u32[2] >> 16);
>>>>>>> +
>>>>>>> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3]
>> &
>>>>>> 0xFFFF;
>>>>>>> + next3 = (dst.u32[3] >> 16);
>>>>>>> +
>>>>>>> + /* Enqueue four to next node */
>>>>>>> + rte_edge_t fix_spec =
>>>>>>> + (next_index ^ next0) | (next_index ^
>> next1) |
>>>>>>> + (next_index ^ next2) | (next_index ^
>> next3);
>>>>>>> +
>>>>>>> + if (unlikely(fix_spec)) {
>>>>>>> + /* Copy things successfully speculated
>> till now
>>>>>> */
>>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>>>> sizeof(from[0]));
>>>>>>> + from += last_spec;
>>>>>>> + to_next += last_spec;
>>>>>>> + held += last_spec;
>>>>>>> + last_spec = 0;
>>>>>>> +
>>>>>>> + /* Next0 */
>>>>>>> + if (next_index == next0) {
>>>>>>> + to_next[0] = from[0];
>>>>>>> + to_next++;
>>>>>>> + held++;
>>>>>>> + } else {
>>>>>>> + rte_node_enqueue_x1(graph,
>> node,
>>>>>> next0,
>>>>>>> + from[0]);
>>>>>>> + }
>>>>>>> +
>>>>>>> + /* Next1 */
>>>>>>> + if (next_index == next1) {
>>>>>>> + to_next[0] = from[1];
>>>>>>> + to_next++;
>>>>>>> + held++;
>>>>>>> + } else {
>>>>>>> + rte_node_enqueue_x1(graph,
>> node,
>>>>>> next1,
>>>>>>> + from[1]);
>>>>>>> + }
>>>>>>> +
>>>>>>> + /* Next2 */
>>>>>>> + if (next_index == next2) {
>>>>>>> + to_next[0] = from[2];
>>>>>>> + to_next++;
>>>>>>> + held++;
>>>>>>> + } else {
>>>>>>> + rte_node_enqueue_x1(graph,
>> node,
>>>>>> next2,
>>>>>>> + from[2]);
>>>>>>> + }
>>>>>>> +
>>>>>>> + /* Next3 */
>>>>>>> + if (next_index == next3) {
>>>>>>> + to_next[0] = from[3];
>>>>>>> + to_next++;
>>>>>>> + held++;
>>>>>>> + } else {
>>>>>>> + rte_node_enqueue_x1(graph,
>> node,
>>>>>> next3,
>>>>>>> + from[3]);
>>>>>>> + }
>>>>>>> +
>>>>>>> + from += 4;
>>>>>>> +
>>>>>>> + } else {
>>>>>>> + last_spec += 4;
>>>>>>> + }
>>>>>>> + }
>>>>>>> +
>>>>>>> + while (n_left_from > 0) {
>>>>>>> + uint32_t next_hop;
>>>>>>> +
>>>>>>> + mbuf0 = pkts[0];
>>>>>>> +
>>>>>>> + pkts += 1;
>>>>>>> + n_left_from -= 1;
>>>>>>> +
>>>>>>> + /* Extract DIP of mbuf0 */
>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>>>> rte_ether_hdr *);
>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum =
>> ipv4_hdr-
>>>>>>> hdr_checksum;
>>>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>>>> time_to_live;
>>>>>>> +
>>>>>>> + rc = rte_lpm_lookup(lpm,
>> rte_be_to_cpu_32(ipv4_hdr-
>>>>>>> dst_addr),
>>>>>>> + &next_hop);
>>>>>>> + next_hop = (rc == 0) ? next_hop : drop_nh;
>>>>>>> +
>>>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop
>> &
>>>>>> 0xFFFF;
>>>>>>> + next0 = (next_hop >> 16);
>>>>>>> +
>>>>>>> + if (unlikely(next_index ^ next0)) {
>>>>>>> + /* Copy things successfully speculated
>> till now
>>>>>> */
>>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>>>> sizeof(from[0]));
>>>>>>> + from += last_spec;
>>>>>>> + to_next += last_spec;
>>>>>>> + held += last_spec;
>>>>>>> + last_spec = 0;
>>>>>>> +
>>>>>>> + rte_node_enqueue_x1(graph, node,
>> next0,
>>>>>> from[0]);
>>>>>>> + from += 1;
>>>>>>> + } else {
>>>>>>> + last_spec += 1;
>>>>>>> + }
>>>>>>> + }
>>>>>>> +
>>>>>>> + /* !!! Home run !!! */
>>>>>>> + if (likely(last_spec == nb_objs)) {
>>>>>>> + rte_node_next_stream_move(graph, node,
>>>>>> next_index);
>>>>>>> + return nb_objs;
>>>>>>> + }
>>>>>>> +
>>>>>>> + held += last_spec;
>>>>>>> + /* Copy things successfully speculated till now */
>>>>>>> + rte_memcpy(to_next, from, last_spec *
>> sizeof(from[0]));
>>>>>>> + rte_node_next_stream_put(graph, node, next_index,
>> held);
>>>>>>> +
>>>>>>> + return nb_objs;
>>>>>>> +}
>>>>>>> +
>>>>>>> #else
>>>>>>>
>>>>>>> static uint16_t
>>>>>>>
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-24 14:38 ` Ray Kinsella
@ 2020-03-26 9:56 ` Pavan Nikhilesh Bhagavatula
2020-03-26 16:54 ` Ray Kinsella
0 siblings, 1 reply; 219+ messages in thread
From: Pavan Nikhilesh Bhagavatula @ 2020-03-26 9:56 UTC (permalink / raw)
To: Ray Kinsella, Jerin Jacob Kollanukkaran, Nithin Kumar Dabilpuram
Cc: dev, thomas, david.marchand, mattias.ronnblom, Kiran Kumar Kokkilagadda
>-----Original Message-----
>From: Ray Kinsella <mdr@ashroe.eu>
>Sent: Tuesday, March 24, 2020 8:08 PM
>To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Jerin
>Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar Dabilpuram
><ndabilpuram@marvell.com>
>Cc: dev@dpdk.org; thomas@monjalon.net;
>david.marchand@redhat.com; mattias.ronnblom@ericsson.com; Kiran
>Kumar Kokkilagadda <kirankumark@marvell.com>
>Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup
>for x86
>
>
>
>On 24/03/2020 09:40, Pavan Nikhilesh Bhagavatula wrote:
>> Hi Ray,
>>
>> I have tried to avoid hand unrolling loops and found the following
>observations.
>>
>> 1. Although it decreases LOC it also takes away readability too.
>> Example:
>> Avoiding unrolled code below
>[SNIP]
>> Which is kind of unreadable.
>
>I am confused - isn't it exactly the same code?
>You still haven't completely unrolled the loop either?
>
>I don't know how one is readable and the other is not.
I guess it’s a matter of personal preference.
>
>>
>> 2. Not all compilers are made equal. I found that most of the
>compilers don’t
>> Unroll the loop above even when compiled with `-funroll-all-loops`.
>> I have checked with following compilers:
>> GCC 9.2.0
>> Clang 9.0.1
>> Aarch64 GCC 7.3.0
>> Aarch64 GCC 9.2.0
>
>Compilers have been unrolling fixed length loops for as long time - this
>isn't new technology.
>
In theory, I agree with your view, but even the latest compiler is not doing a decent
job on unrolling the loop.
We can revisit this scheme, if and when compiler smart enough to do this as just unrolling
the loops is not good enough. It has to do it better than hand unrolling.
For example on arm64 GCC doesn’t merge load/stores to load/store pairs when unrolling.
(both ldr/str and ldp/stp latency is 3cyc and effectively ldp cost is halved).
Even on x86 we see extra ~100 cycles as mentioned in [1].
>If the compiler isn't unrolling you are doing something that makes it
>think it is a bad idea.
>Hand unrolling the loop isn't the solution, understanding what the
>compiler is doing is a better idea.
>
>In front of your for loop insert, to indicate to the compiler what you
>want to do.
>#pragma unroll BUF_PER_LOOP
Can you check which versions of compiler which this pragma works on?
Most of the gcc versions that I have tried just spit out the following warning.
../lib/librte_node/ip4_rewrite.c:59: warning: ignoring #pragma unroll BUF_PER_LOOP [-Wunknown-pragmas]
59 | #pragma unroll BUF_PER_LOOP
>
>With clang you can ask it why it is not unrolling the loop with the
>following switches.
>(output is verbose, but the reason is in there).
>
>-Rpass=loop-unroll -Rpass-missed=loop-unroll
>
../lib/librte_node/ip4_rewrite.c:57:2: remark: unrolled loop by a factor of 4 with run-time trip count [-Rpass=loop-unroll]
for (i = 0; i < nb_objs; i++) {
^
>>
>> 3. Performance wise I see a lot of degradation on our platform at least
>13%.
>
>Is the loop being unrolled?
>
Yes, in a suboptimal way. https://pastebin.com/nkXvzMiW
we decide to stick with hand unrolling based on the test result instead of depending on compilers mercy to do it for us[2].
if you think, it can be improved then please submit a patch.
[2]https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88760
>> On IA with a Broadwell(Xeon E5-2690) and i40e the performance
>remain same w.r.t Rx/Tx since the
>> hotspot is in the Tx path of the driver which limits the per core
>capability.
>> But the performance difference in number of cycles per node can
>be seen below:
>>
[1]
>> Hand unrolling:
>> +-------------------------------+---------------+---------------+---------------+-
>--------------+---------------+-----------+
>> |Node |calls |objs |realloc_count |objs/call
>|objs/sec(10E6) |cycles/call|
>> +-------------------------------+---------------+---------------+---------------+-
>--------------+---------------+-----------+
>> |ip4_lookup |7765918 |248509344 |1 |32.000
>|27.725408 |779.0000 |
>> |ip4_rewrite |7765925 |248509568 |1 |32.000
>|27.725408 |425.0000 |
>> |ethdev_tx-1 |7765927 |204056223 |1 |26.000
>|22.762720 |597.0000 |
>> |pkt_drop |1389170 |44453409 |1 |32.000
>|4.962688 |298.0000 |
>> |ethdev_rx-0-0 |63604111 |248509792 |2 |32.000
>|27.725408 |982.0000 |
>> +-------------------------------+---------------+---------------+---------------+-
>--------------+---------------+-----------+
>>
>> W/o unrolling:
>>
>> +-------------------------------+---------------+---------------+---------------+-
>--------------+---------------+-----------+
>> |Node |calls |objs |realloc_count |objs/call
>|objs/sec(10E6) |cycles/call|
>> +-------------------------------+---------------+---------------+---------------+-
>--------------+---------------+-----------+
>> |ip4_lookup |18864640 |603668448 |1 |32.000
>|26.051328 |828.0000 |
>> |ip4_rewrite |18864646 |603668640 |1 |32.000
>|26.051328 |534.0000 |
>> |ethdev_tx-1 |18864648 |527874175 |1 |27.000
>|22.780256 |633.0000 |
>> |pkt_drop |2368580 |75794529 |1 |32.000
>|3.271072 |286.0000 |
>> |ethdev_rx-0-0 |282058226 |603668864 |2 |32.000
>|26.051328 |994.0000 |
>> +-------------------------------+---------------+---------------+---------------+-
>--------------+---------------+-----------+
>>
>> Considering the above findings we would like to continue unrolling
>the loops by hand.
>>
>> Regards,
>> Pavan.
>>
>>> -----Original Message-----
>>> From: Ray Kinsella <mdr@ashroe.eu>
>>> Sent: Friday, March 20, 2020 2:44 PM
>>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>Jerin
>>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar
>Dabilpuram
>>> <ndabilpuram@marvell.com>
>>> Cc: dev@dpdk.org; thomas@monjalon.net;
>>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com;
>Kiran
>>> Kumar Kokkilagadda <kirankumark@marvell.com>
>>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4
>lookup
>>> for x86
>>>
>>>
>>>
>>> On 19/03/2020 16:13, Pavan Nikhilesh Bhagavatula wrote:
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Ray Kinsella <mdr@ashroe.eu>
>>>>> Sent: Thursday, March 19, 2020 9:21 PM
>>>>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>>> Jerin
>>>>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar
>>> Dabilpuram
>>>>> <ndabilpuram@marvell.com>
>>>>> Cc: dev@dpdk.org; thomas@monjalon.net;
>>>>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com;
>>> Kiran
>>>>> Kumar Kokkilagadda <kirankumark@marvell.com>
>>>>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4
>>> lookup
>>>>> for x86
>>>>>
>>>>>
>>>>>
>>>>> On 19/03/2020 14:22, Pavan Nikhilesh Bhagavatula wrote:
>>>>>>> On 18/03/2020 21:35, jerinj@marvell.com wrote:
>>>>>>>> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>>>>
>>>>>>>> Add IPv4 lookup process function for ip4_lookup
>>>>>>>> rte_node. This node performs LPM lookup using x86_64
>>>>>>>> vector supported RTE_LPM API on every packet received
>>>>>>>> and forwards it to a next node that is identified by
>>>>>>>> lookup result.
>>>>>>>>
>>>>>>>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>>>> Signed-off-by: Nithin Dabilpuram
><ndabilpuram@marvell.com>
>>>>>>>> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
>>>>>>>> ---
>>>>>>>> lib/librte_node/ip4_lookup.c | 245
>>>>>>> +++++++++++++++++++++++++++++++++++
>>>>>>>> 1 file changed, 245 insertions(+)
>>>>>>>>
>>>>>>>> diff --git a/lib/librte_node/ip4_lookup.c
>>>>>>> b/lib/librte_node/ip4_lookup.c
>>>>>>>> index d7fcd1158..c003e9c91 100644
>>>>>>>> --- a/lib/librte_node/ip4_lookup.c
>>>>>>>> +++ b/lib/librte_node/ip4_lookup.c
>>>>>>>> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct
>>>>> rte_graph
>>>>>>> *graph, struct rte_node *node,
>>>>>>>> return nb_objs;
>>>>>>>> }
>>>>>>>>
>>>>>>>> +#elif defined(RTE_ARCH_X86)
>>>>>>>> +
>>>>>>>> +/* X86 SSE */
>>>>>>>> +static uint16_t
>>>>>>>> +ip4_lookup_node_process(struct rte_graph *graph, struct
>>>>> rte_node
>>>>>>> *node,
>>>>>>>> + void **objs, uint16_t nb_objs)
>>>>>>>> +{
>>>>>>>> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3,
>>> **pkts;
>>>>>>>> + rte_edge_t next0, next1, next2, next3, next_index;
>>>>>>>> + struct rte_ipv4_hdr *ipv4_hdr;
>>>>>>>> + struct rte_ether_hdr *eth_hdr;
>>>>>>>> + uint32_t ip0, ip1, ip2, ip3;
>>>>>>>> + void **to_next, **from;
>>>>>>>> + uint16_t last_spec = 0;
>>>>>>>> + uint16_t n_left_from;
>>>>>>>> + struct rte_lpm *lpm;
>>>>>>>> + uint16_t held = 0;
>>>>>>>> + uint32_t drop_nh;
>>>>>>>> + rte_xmm_t dst;
>>>>>>>> + __m128i dip; /* SSE register */
>>>>>>>> + int rc, i;
>>>>>>>> +
>>>>>>>> + /* Speculative next */
>>>>>>>> + next_index =
>>> RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
>>>>>>>> + /* Drop node */
>>>>>>>> + drop_nh =
>>>>>>> ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
>>>>>>>> +
>>>>>>>> + /* Get socket specific LPM from ctx */
>>>>>>>> + lpm = *((struct rte_lpm **)node->ctx);
>>>>>>>> +
>>>>>>>> + pkts = (struct rte_mbuf **)objs;
>>>>>>>> + from = objs;
>>>>>>>> + n_left_from = nb_objs;
>>>>>>>
>>>>>>> I doubt this initial prefetch of the first 4 packets has any
>benefit.
>>>>>>
>>>>>> Ack will remove in v2 for x86.
>>>>>>
>>>>>>>
>>>>>>>> + if (n_left_from >= 4) {
>>>>>>>> + for (i = 0; i < 4; i++) {
>>>>>>>> +
>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
>>>>>>>> + struct
>>> rte_ether_hdr
>>>>>>> *) +
>>>>>>>> + 1);
>>>>>>>> + }
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + /* Get stream for the speculated next node */
>>>>>>>> + to_next = rte_node_next_stream_get(graph, node,
>>>>>>> next_index, nb_objs);
>>>>>>>
>>>>>>> Suggest you don't reuse the hand-unrolling optimization from
>>> FD.io
>>>>>>> VPP.
>>>>>>> I have never found any performance benefit from them, and
>they
>>>>>>> make the code unnecessarily verbose.
>>>>>>>
>>>>>>
>>>>>> How would be take the benefit of rte_lpm_lookupx4 without
>>>>> unrolling the loop?.
>>>>>> Also, in future if we are using rte_rib and fib with a CPU
>supporting
>>>>> wider SIMD we might
>>>>>> need to unroll them further (AVX256 AND 512 currently
>>>>> rte_lpm_lookup uses only 128bit
>>>>>> since it is only uses SSE extension).
>>>>>
>>>>> Let the compiler do it for you, but using a constant vector length.
>>>>> for (int i=0; i < 4; ++i) { ... }
>>>>>
>>>>
>>>> Ok, I think I misunderstood the previous comment.
>>>> It was only for the prefetches in the loop right?
>>>
>>>
>>> no, it was for all the needless repetition.
>>> hand-unrolling loops serve no purpose but to add verbosity.
>>>
>>>>
>>>>>>
>>>>>>>
>>>>>>>> + while (n_left_from >= 4) {
>>>>>>>> + /* Prefetch next-next mbufs */
>>>>>>>> + if (likely(n_left_from >= 11)) {
>>>>>>>> + rte_prefetch0(pkts[8]);
>>>>>>>> + rte_prefetch0(pkts[9]);
>>>>>>>> + rte_prefetch0(pkts[10]);
>>>>>>>> + rte_prefetch0(pkts[11]);
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + /* Prefetch next mbuf data */
>>>>>>>> + if (likely(n_left_from >= 7)) {
>>>>>>>> +
>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
>>>>>>>> + struct
>>> rte_ether_hdr
>>>>>>> *) +
>>>>>>>> + 1);
>>>>>>>> +
>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
>>>>>>>> + struct
>>> rte_ether_hdr
>>>>>>> *) +
>>>>>>>> + 1);
>>>>>>>> +
>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
>>>>>>>> + struct
>>> rte_ether_hdr
>>>>>>> *) +
>>>>>>>> + 1);
>>>>>>>> +
>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
>>>>>>>> + struct
>>> rte_ether_hdr
>>>>>>> *) +
>>>>>>>> + 1);
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + mbuf0 = pkts[0];
>>>>>>>> + mbuf1 = pkts[1];
>>>>>>>> + mbuf2 = pkts[2];
>>>>>>>> + mbuf3 = pkts[3];
>>>>>>>> +
>>>>>>>> + pkts += 4;
>>>>>>>> + n_left_from -= 4;
>>>>>>>> +
>>>>>>>> + /* Extract DIP of mbuf0 */
>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>>>>> rte_ether_hdr *);
>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>> + ip0 = ipv4_hdr->dst_addr;
>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum =
>>> ipv4_hdr-
>>>>>>>> hdr_checksum;
>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>>>>> time_to_live;
>>>>>>>> +
>>>>>>>> + /* Extract DIP of mbuf1 */
>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct
>>>>>>> rte_ether_hdr *);
>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>> + ip1 = ipv4_hdr->dst_addr;
>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>> + rte_node_mbuf_priv1(mbuf1)->cksum =
>>> ipv4_hdr-
>>>>>>>> hdr_checksum;
>>>>>>>> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-
>>>>>>>> time_to_live;
>>>>>>>> +
>>>>>>>> + /* Extract DIP of mbuf2 */
>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct
>>>>>>> rte_ether_hdr *);
>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>> + ip2 = ipv4_hdr->dst_addr;
>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>> + rte_node_mbuf_priv1(mbuf2)->cksum =
>>> ipv4_hdr-
>>>>>>>> hdr_checksum;
>>>>>>>> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-
>>>>>>>> time_to_live;
>>>>>>>> +
>>>>>>>> + /* Extract DIP of mbuf3 */
>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct
>>>>>>> rte_ether_hdr *);
>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>> + ip3 = ipv4_hdr->dst_addr;
>>>>>>>> +
>>>>>>>> + /* Prepare for lookup x4 */
>>>>>>>> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
>>>>>>>> +
>>>>>>>> + /* Byte swap 4 IPV4 addresses. */
>>>>>>>> + const __m128i bswap_mask = _mm_set_epi8(
>>>>>>>> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1,
>>> 2, 3);
>>>>>>>> + dip = _mm_shuffle_epi8(dip, bswap_mask);
>>>>>>>> +
>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>> + rte_node_mbuf_priv1(mbuf3)->cksum =
>>> ipv4_hdr-
>>>>>>>> hdr_checksum;
>>>>>>>> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-
>>>>>>>> time_to_live;
>>>>>>>> +
>>>>>>>> + /* Perform LPM lookup to get NH and next
>>> node */
>>>>>>>> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
>>>>>>>> +
>>>>>>>> + /* Extract next node id and NH */
>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0]
>>> &
>>>>>>> 0xFFFF;
>>>>>>>> + next0 = (dst.u32[0] >> 16);
>>>>>>>> +
>>>>>>>> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1]
>>> &
>>>>>>> 0xFFFF;
>>>>>>>> + next1 = (dst.u32[1] >> 16);
>>>>>>>> +
>>>>>>>> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2]
>>> &
>>>>>>> 0xFFFF;
>>>>>>>> + next2 = (dst.u32[2] >> 16);
>>>>>>>> +
>>>>>>>> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3]
>>> &
>>>>>>> 0xFFFF;
>>>>>>>> + next3 = (dst.u32[3] >> 16);
>>>>>>>> +
>>>>>>>> + /* Enqueue four to next node */
>>>>>>>> + rte_edge_t fix_spec =
>>>>>>>> + (next_index ^ next0) | (next_index ^
>>> next1) |
>>>>>>>> + (next_index ^ next2) | (next_index ^
>>> next3);
>>>>>>>> +
>>>>>>>> + if (unlikely(fix_spec)) {
>>>>>>>> + /* Copy things successfully speculated
>>> till now
>>>>>>> */
>>>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>>>>> sizeof(from[0]));
>>>>>>>> + from += last_spec;
>>>>>>>> + to_next += last_spec;
>>>>>>>> + held += last_spec;
>>>>>>>> + last_spec = 0;
>>>>>>>> +
>>>>>>>> + /* Next0 */
>>>>>>>> + if (next_index == next0) {
>>>>>>>> + to_next[0] = from[0];
>>>>>>>> + to_next++;
>>>>>>>> + held++;
>>>>>>>> + } else {
>>>>>>>> + rte_node_enqueue_x1(graph,
>>> node,
>>>>>>> next0,
>>>>>>>> + from[0]);
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + /* Next1 */
>>>>>>>> + if (next_index == next1) {
>>>>>>>> + to_next[0] = from[1];
>>>>>>>> + to_next++;
>>>>>>>> + held++;
>>>>>>>> + } else {
>>>>>>>> + rte_node_enqueue_x1(graph,
>>> node,
>>>>>>> next1,
>>>>>>>> + from[1]);
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + /* Next2 */
>>>>>>>> + if (next_index == next2) {
>>>>>>>> + to_next[0] = from[2];
>>>>>>>> + to_next++;
>>>>>>>> + held++;
>>>>>>>> + } else {
>>>>>>>> + rte_node_enqueue_x1(graph,
>>> node,
>>>>>>> next2,
>>>>>>>> + from[2]);
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + /* Next3 */
>>>>>>>> + if (next_index == next3) {
>>>>>>>> + to_next[0] = from[3];
>>>>>>>> + to_next++;
>>>>>>>> + held++;
>>>>>>>> + } else {
>>>>>>>> + rte_node_enqueue_x1(graph,
>>> node,
>>>>>>> next3,
>>>>>>>> + from[3]);
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + from += 4;
>>>>>>>> +
>>>>>>>> + } else {
>>>>>>>> + last_spec += 4;
>>>>>>>> + }
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + while (n_left_from > 0) {
>>>>>>>> + uint32_t next_hop;
>>>>>>>> +
>>>>>>>> + mbuf0 = pkts[0];
>>>>>>>> +
>>>>>>>> + pkts += 1;
>>>>>>>> + n_left_from -= 1;
>>>>>>>> +
>>>>>>>> + /* Extract DIP of mbuf0 */
>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>>>>> rte_ether_hdr *);
>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum =
>>> ipv4_hdr-
>>>>>>>> hdr_checksum;
>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>>>>> time_to_live;
>>>>>>>> +
>>>>>>>> + rc = rte_lpm_lookup(lpm,
>>> rte_be_to_cpu_32(ipv4_hdr-
>>>>>>>> dst_addr),
>>>>>>>> + &next_hop);
>>>>>>>> + next_hop = (rc == 0) ? next_hop : drop_nh;
>>>>>>>> +
>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop
>>> &
>>>>>>> 0xFFFF;
>>>>>>>> + next0 = (next_hop >> 16);
>>>>>>>> +
>>>>>>>> + if (unlikely(next_index ^ next0)) {
>>>>>>>> + /* Copy things successfully speculated
>>> till now
>>>>>>> */
>>>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>>>>> sizeof(from[0]));
>>>>>>>> + from += last_spec;
>>>>>>>> + to_next += last_spec;
>>>>>>>> + held += last_spec;
>>>>>>>> + last_spec = 0;
>>>>>>>> +
>>>>>>>> + rte_node_enqueue_x1(graph, node,
>>> next0,
>>>>>>> from[0]);
>>>>>>>> + from += 1;
>>>>>>>> + } else {
>>>>>>>> + last_spec += 1;
>>>>>>>> + }
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + /* !!! Home run !!! */
>>>>>>>> + if (likely(last_spec == nb_objs)) {
>>>>>>>> + rte_node_next_stream_move(graph, node,
>>>>>>> next_index);
>>>>>>>> + return nb_objs;
>>>>>>>> + }
>>>>>>>> +
>>>>>>>> + held += last_spec;
>>>>>>>> + /* Copy things successfully speculated till now */
>>>>>>>> + rte_memcpy(to_next, from, last_spec *
>>> sizeof(from[0]));
>>>>>>>> + rte_node_next_stream_put(graph, node, next_index,
>>> held);
>>>>>>>> +
>>>>>>>> + return nb_objs;
>>>>>>>> +}
>>>>>>>> +
>>>>>>>> #else
>>>>>>>>
>>>>>>>> static uint16_t
>>>>>>>>
^ permalink raw reply [flat|nested] 219+ messages in thread
* Re: [dpdk-dev] [EXT] Re: [PATCH v1 20/26] node: ipv4 lookup for x86
2020-03-26 9:56 ` Pavan Nikhilesh Bhagavatula
@ 2020-03-26 16:54 ` Ray Kinsella
0 siblings, 0 replies; 219+ messages in thread
From: Ray Kinsella @ 2020-03-26 16:54 UTC (permalink / raw)
To: Pavan Nikhilesh Bhagavatula, Jerin Jacob Kollanukkaran,
Nithin Kumar Dabilpuram
Cc: dev, thomas, david.marchand, mattias.ronnblom, Kiran Kumar Kokkilagadda
On 26/03/2020 09:56, Pavan Nikhilesh Bhagavatula wrote:
>
>> -----Original Message-----
>> From: Ray Kinsella <mdr@ashroe.eu>
>> Sent: Tuesday, March 24, 2020 8:08 PM
>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>; Jerin
>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar Dabilpuram
>> <ndabilpuram@marvell.com>
>> Cc: dev@dpdk.org; thomas@monjalon.net;
>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com; Kiran
>> Kumar Kokkilagadda <kirankumark@marvell.com>
>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4 lookup
>> for x86
>>
>>
>>
>> On 24/03/2020 09:40, Pavan Nikhilesh Bhagavatula wrote:
>>> Hi Ray,
>>>
>>> I have tried to avoid hand unrolling loops and found the following
>> observations.
>>>
>>> 1. Although it decreases LOC it also takes away readability too.
>>> Example:
>>> Avoiding unrolled code below
>> [SNIP]
>>> Which is kind of unreadable.
>>
>> I am confused - isn't it exactly the same code?
>> You still haven't completely unrolled the loop either?
>>
>> I don't know how one is readable and the other is not.
>
> I guess it’s a matter of personal preference.
You have a rare preference for verbosity, sir.
>
>>
>>>
>>> 2. Not all compilers are made equal. I found that most of the
>> compilers don’t
>>> Unroll the loop above even when compiled with `-funroll-all-loops`.
>>> I have checked with following compilers:
>>> GCC 9.2.0
>>> Clang 9.0.1
>>> Aarch64 GCC 7.3.0
>>> Aarch64 GCC 9.2.0
>>
>> Compilers have been unrolling fixed length loops for as long time - this
>> isn't new technology.
>>
>
> In theory, I agree with your view, but even the latest compiler is not doing a decent
> job on unrolling the loop.
> We can revisit this scheme, if and when compiler smart enough to do this as just unrolling
> the loops is not good enough. It has to do it better than hand unrolling.
> For example on arm64 GCC doesn’t merge load/stores to load/store pairs when unrolling.
> (both ldr/str and ldp/stp latency is 3cyc and effectively ldp cost is halved).
>
> Even on x86 we see extra ~100 cycles as mentioned in [1].
>
>> If the compiler isn't unrolling you are doing something that makes it
>> think it is a bad idea.
>> Hand unrolling the loop isn't the solution, understanding what the
>> compiler is doing is a better idea.
>>
>> In front of your for loop insert, to indicate to the compiler what you
>> want to do.
>> #pragma unroll BUF_PER_LOOP
>
> Can you check which versions of compiler which this pragma works on?
https://gcc.gnu.org/onlinedocs/gcc/Loop-Specific-Pragmas.html
https://clang.llvm.org/docs/AttributeReference.htm
>
> Most of the gcc versions that I have tried just spit out the following warning.
> ../lib/librte_node/ip4_rewrite.c:59: warning: ignoring #pragma unroll BUF_PER_LOOP [-Wunknown-pragmas]
> 59 | #pragma unroll BUF_PER_LOOP
Not sure on this one ...
>>
>> With clang you can ask it why it is not unrolling the loop with the
>> following switches.
>> (output is verbose, but the reason is in there).
>>
>> -Rpass=loop-unroll -Rpass-missed=loop-unroll
>>
>
> ../lib/librte_node/ip4_rewrite.c:57:2: remark: unrolled loop by a factor of 4 with run-time trip count [-Rpass=loop-unroll]
> for (i = 0; i < nb_objs; i++) {
> ^
That is good ...
>>>
>>> 3. Performance wise I see a lot of degradation on our platform at least
>> 13%.
>>
>> Is the loop being unrolled?
>>
>
> Yes, in a suboptimal way. https://pastebin.com/nkXvzMiW
>
> we decide to stick with hand unrolling based on the test result instead of depending on compilers mercy to do it for us[2].
Why not code in assembler then, taking to it logical conclusion.
> if you think, it can be improved then please submit a patch.
I don't have an ARM platform to test on.
From my own work in FD.io VPP, which you are clear borrowing heavily from, to put it kindly.
I have never seen an advantage from hand-unrolling compared to letting the compiler do it.
Perhaps this case is the exception.
I would be suspicious though that the compiler is making bad decisions for some reason.
In anycase, if you (and the presumabily the maintainer) are happy with verbose code like this making it's way into the codebase.
I won't argue with it.
> [2]https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88760
>
>>> On IA with a Broadwell(Xeon E5-2690) and i40e the performance
>> remain same w.r.t Rx/Tx since the
>>> hotspot is in the Tx path of the driver which limits the per core
>> capability.
>>> But the performance difference in number of cycles per node can
>> be seen below:
>>>
>
> [1]
>
>>> Hand unrolling:
>>> +-------------------------------+---------------+---------------+---------------+-
>> --------------+---------------+-----------+
>>> |Node |calls |objs |realloc_count |objs/call
>> |objs/sec(10E6) |cycles/call|
>>> +-------------------------------+---------------+---------------+---------------+-
>> --------------+---------------+-----------+
>>> |ip4_lookup |7765918 |248509344 |1 |32.000
>> |27.725408 |779.0000 |
>>> |ip4_rewrite |7765925 |248509568 |1 |32.000
>> |27.725408 |425.0000 |
>>> |ethdev_tx-1 |7765927 |204056223 |1 |26.000
>> |22.762720 |597.0000 |
>>> |pkt_drop |1389170 |44453409 |1 |32.000
>> |4.962688 |298.0000 |
>>> |ethdev_rx-0-0 |63604111 |248509792 |2 |32.000
>> |27.725408 |982.0000 |
>>> +-------------------------------+---------------+---------------+---------------+-
>> --------------+---------------+-----------+
>>>
>>> W/o unrolling:
>>>
>>> +-------------------------------+---------------+---------------+---------------+-
>> --------------+---------------+-----------+
>>> |Node |calls |objs |realloc_count |objs/call
>> |objs/sec(10E6) |cycles/call|
>>> +-------------------------------+---------------+---------------+---------------+-
>> --------------+---------------+-----------+
>>> |ip4_lookup |18864640 |603668448 |1 |32.000
>> |26.051328 |828.0000 |
>>> |ip4_rewrite |18864646 |603668640 |1 |32.000
>> |26.051328 |534.0000 |
>>> |ethdev_tx-1 |18864648 |527874175 |1 |27.000
>> |22.780256 |633.0000 |
>>> |pkt_drop |2368580 |75794529 |1 |32.000
>> |3.271072 |286.0000 |
>>> |ethdev_rx-0-0 |282058226 |603668864 |2 |32.000
>> |26.051328 |994.0000 |
>>> +-------------------------------+---------------+---------------+---------------+-
>> --------------+---------------+-----------+
>>>
>>> Considering the above findings we would like to continue unrolling
>> the loops by hand.
>>>
>>> Regards,
>>> Pavan.
>>>
>>>> -----Original Message-----
>>>> From: Ray Kinsella <mdr@ashroe.eu>
>>>> Sent: Friday, March 20, 2020 2:44 PM
>>>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>> Jerin
>>>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar
>> Dabilpuram
>>>> <ndabilpuram@marvell.com>
>>>> Cc: dev@dpdk.org; thomas@monjalon.net;
>>>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com;
>> Kiran
>>>> Kumar Kokkilagadda <kirankumark@marvell.com>
>>>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4
>> lookup
>>>> for x86
>>>>
>>>>
>>>>
>>>> On 19/03/2020 16:13, Pavan Nikhilesh Bhagavatula wrote:
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Ray Kinsella <mdr@ashroe.eu>
>>>>>> Sent: Thursday, March 19, 2020 9:21 PM
>>>>>> To: Pavan Nikhilesh Bhagavatula <pbhagavatula@marvell.com>;
>>>> Jerin
>>>>>> Jacob Kollanukkaran <jerinj@marvell.com>; Nithin Kumar
>>>> Dabilpuram
>>>>>> <ndabilpuram@marvell.com>
>>>>>> Cc: dev@dpdk.org; thomas@monjalon.net;
>>>>>> david.marchand@redhat.com; mattias.ronnblom@ericsson.com;
>>>> Kiran
>>>>>> Kumar Kokkilagadda <kirankumark@marvell.com>
>>>>>> Subject: Re: [EXT] Re: [dpdk-dev] [PATCH v1 20/26] node: ipv4
>>>> lookup
>>>>>> for x86
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 19/03/2020 14:22, Pavan Nikhilesh Bhagavatula wrote:
>>>>>>>> On 18/03/2020 21:35, jerinj@marvell.com wrote:
>>>>>>>>> From: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>>>>>
>>>>>>>>> Add IPv4 lookup process function for ip4_lookup
>>>>>>>>> rte_node. This node performs LPM lookup using x86_64
>>>>>>>>> vector supported RTE_LPM API on every packet received
>>>>>>>>> and forwards it to a next node that is identified by
>>>>>>>>> lookup result.
>>>>>>>>>
>>>>>>>>> Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
>>>>>>>>> Signed-off-by: Nithin Dabilpuram
>> <ndabilpuram@marvell.com>
>>>>>>>>> Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
>>>>>>>>> ---
>>>>>>>>> lib/librte_node/ip4_lookup.c | 245
>>>>>>>> +++++++++++++++++++++++++++++++++++
>>>>>>>>> 1 file changed, 245 insertions(+)
>>>>>>>>>
>>>>>>>>> diff --git a/lib/librte_node/ip4_lookup.c
>>>>>>>> b/lib/librte_node/ip4_lookup.c
>>>>>>>>> index d7fcd1158..c003e9c91 100644
>>>>>>>>> --- a/lib/librte_node/ip4_lookup.c
>>>>>>>>> +++ b/lib/librte_node/ip4_lookup.c
>>>>>>>>> @@ -264,6 +264,251 @@ ip4_lookup_node_process(struct
>>>>>> rte_graph
>>>>>>>> *graph, struct rte_node *node,
>>>>>>>>> return nb_objs;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> +#elif defined(RTE_ARCH_X86)
>>>>>>>>> +
>>>>>>>>> +/* X86 SSE */
>>>>>>>>> +static uint16_t
>>>>>>>>> +ip4_lookup_node_process(struct rte_graph *graph, struct
>>>>>> rte_node
>>>>>>>> *node,
>>>>>>>>> + void **objs, uint16_t nb_objs)
>>>>>>>>> +{
>>>>>>>>> + struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3,
>>>> **pkts;
>>>>>>>>> + rte_edge_t next0, next1, next2, next3, next_index;
>>>>>>>>> + struct rte_ipv4_hdr *ipv4_hdr;
>>>>>>>>> + struct rte_ether_hdr *eth_hdr;
>>>>>>>>> + uint32_t ip0, ip1, ip2, ip3;
>>>>>>>>> + void **to_next, **from;
>>>>>>>>> + uint16_t last_spec = 0;
>>>>>>>>> + uint16_t n_left_from;
>>>>>>>>> + struct rte_lpm *lpm;
>>>>>>>>> + uint16_t held = 0;
>>>>>>>>> + uint32_t drop_nh;
>>>>>>>>> + rte_xmm_t dst;
>>>>>>>>> + __m128i dip; /* SSE register */
>>>>>>>>> + int rc, i;
>>>>>>>>> +
>>>>>>>>> + /* Speculative next */
>>>>>>>>> + next_index =
>>>> RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
>>>>>>>>> + /* Drop node */
>>>>>>>>> + drop_nh =
>>>>>>>> ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
>>>>>>>>> +
>>>>>>>>> + /* Get socket specific LPM from ctx */
>>>>>>>>> + lpm = *((struct rte_lpm **)node->ctx);
>>>>>>>>> +
>>>>>>>>> + pkts = (struct rte_mbuf **)objs;
>>>>>>>>> + from = objs;
>>>>>>>>> + n_left_from = nb_objs;
>>>>>>>>
>>>>>>>> I doubt this initial prefetch of the first 4 packets has any
>> benefit.
>>>>>>>
>>>>>>> Ack will remove in v2 for x86.
>>>>>>>
>>>>>>>>
>>>>>>>>> + if (n_left_from >= 4) {
>>>>>>>>> + for (i = 0; i < 4; i++) {
>>>>>>>>> +
>>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[i],
>>>>>>>>> + struct
>>>> rte_ether_hdr
>>>>>>>> *) +
>>>>>>>>> + 1);
>>>>>>>>> + }
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + /* Get stream for the speculated next node */
>>>>>>>>> + to_next = rte_node_next_stream_get(graph, node,
>>>>>>>> next_index, nb_objs);
>>>>>>>>
>>>>>>>> Suggest you don't reuse the hand-unrolling optimization from
>>>> FD.io
>>>>>>>> VPP.
>>>>>>>> I have never found any performance benefit from them, and
>> they
>>>>>>>> make the code unnecessarily verbose.
>>>>>>>>
>>>>>>>
>>>>>>> How would be take the benefit of rte_lpm_lookupx4 without
>>>>>> unrolling the loop?.
>>>>>>> Also, in future if we are using rte_rib and fib with a CPU
>> supporting
>>>>>> wider SIMD we might
>>>>>>> need to unroll them further (AVX256 AND 512 currently
>>>>>> rte_lpm_lookup uses only 128bit
>>>>>>> since it is only uses SSE extension).
>>>>>>
>>>>>> Let the compiler do it for you, but using a constant vector length.
>>>>>> for (int i=0; i < 4; ++i) { ... }
>>>>>>
>>>>>
>>>>> Ok, I think I misunderstood the previous comment.
>>>>> It was only for the prefetches in the loop right?
>>>>
>>>>
>>>> no, it was for all the needless repetition.
>>>> hand-unrolling loops serve no purpose but to add verbosity.
>>>>
>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> + while (n_left_from >= 4) {
>>>>>>>>> + /* Prefetch next-next mbufs */
>>>>>>>>> + if (likely(n_left_from >= 11)) {
>>>>>>>>> + rte_prefetch0(pkts[8]);
>>>>>>>>> + rte_prefetch0(pkts[9]);
>>>>>>>>> + rte_prefetch0(pkts[10]);
>>>>>>>>> + rte_prefetch0(pkts[11]);
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + /* Prefetch next mbuf data */
>>>>>>>>> + if (likely(n_left_from >= 7)) {
>>>>>>>>> +
>>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[4],
>>>>>>>>> + struct
>>>> rte_ether_hdr
>>>>>>>> *) +
>>>>>>>>> + 1);
>>>>>>>>> +
>>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[5],
>>>>>>>>> + struct
>>>> rte_ether_hdr
>>>>>>>> *) +
>>>>>>>>> + 1);
>>>>>>>>> +
>>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[6],
>>>>>>>>> + struct
>>>> rte_ether_hdr
>>>>>>>> *) +
>>>>>>>>> + 1);
>>>>>>>>> +
>>>> rte_prefetch0(rte_pktmbuf_mtod(pkts[7],
>>>>>>>>> + struct
>>>> rte_ether_hdr
>>>>>>>> *) +
>>>>>>>>> + 1);
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + mbuf0 = pkts[0];
>>>>>>>>> + mbuf1 = pkts[1];
>>>>>>>>> + mbuf2 = pkts[2];
>>>>>>>>> + mbuf3 = pkts[3];
>>>>>>>>> +
>>>>>>>>> + pkts += 4;
>>>>>>>>> + n_left_from -= 4;
>>>>>>>>> +
>>>>>>>>> + /* Extract DIP of mbuf0 */
>>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>>>>>> rte_ether_hdr *);
>>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>>> + ip0 = ipv4_hdr->dst_addr;
>>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum =
>>>> ipv4_hdr-
>>>>>>>>> hdr_checksum;
>>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>>>>>> time_to_live;
>>>>>>>>> +
>>>>>>>>> + /* Extract DIP of mbuf1 */
>>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf1, struct
>>>>>>>> rte_ether_hdr *);
>>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>>> + ip1 = ipv4_hdr->dst_addr;
>>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>>> + rte_node_mbuf_priv1(mbuf1)->cksum =
>>>> ipv4_hdr-
>>>>>>>>> hdr_checksum;
>>>>>>>>> + rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr-
>>>>>>>>> time_to_live;
>>>>>>>>> +
>>>>>>>>> + /* Extract DIP of mbuf2 */
>>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf2, struct
>>>>>>>> rte_ether_hdr *);
>>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>>> + ip2 = ipv4_hdr->dst_addr;
>>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>>> + rte_node_mbuf_priv1(mbuf2)->cksum =
>>>> ipv4_hdr-
>>>>>>>>> hdr_checksum;
>>>>>>>>> + rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr-
>>>>>>>>> time_to_live;
>>>>>>>>> +
>>>>>>>>> + /* Extract DIP of mbuf3 */
>>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf3, struct
>>>>>>>> rte_ether_hdr *);
>>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>>> + ip3 = ipv4_hdr->dst_addr;
>>>>>>>>> +
>>>>>>>>> + /* Prepare for lookup x4 */
>>>>>>>>> + dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
>>>>>>>>> +
>>>>>>>>> + /* Byte swap 4 IPV4 addresses. */
>>>>>>>>> + const __m128i bswap_mask = _mm_set_epi8(
>>>>>>>>> + 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1,
>>>> 2, 3);
>>>>>>>>> + dip = _mm_shuffle_epi8(dip, bswap_mask);
>>>>>>>>> +
>>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>>> + rte_node_mbuf_priv1(mbuf3)->cksum =
>>>> ipv4_hdr-
>>>>>>>>> hdr_checksum;
>>>>>>>>> + rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr-
>>>>>>>>> time_to_live;
>>>>>>>>> +
>>>>>>>>> + /* Perform LPM lookup to get NH and next
>>>> node */
>>>>>>>>> + rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
>>>>>>>>> +
>>>>>>>>> + /* Extract next node id and NH */
>>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0]
>>>> &
>>>>>>>> 0xFFFF;
>>>>>>>>> + next0 = (dst.u32[0] >> 16);
>>>>>>>>> +
>>>>>>>>> + rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1]
>>>> &
>>>>>>>> 0xFFFF;
>>>>>>>>> + next1 = (dst.u32[1] >> 16);
>>>>>>>>> +
>>>>>>>>> + rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2]
>>>> &
>>>>>>>> 0xFFFF;
>>>>>>>>> + next2 = (dst.u32[2] >> 16);
>>>>>>>>> +
>>>>>>>>> + rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3]
>>>> &
>>>>>>>> 0xFFFF;
>>>>>>>>> + next3 = (dst.u32[3] >> 16);
>>>>>>>>> +
>>>>>>>>> + /* Enqueue four to next node */
>>>>>>>>> + rte_edge_t fix_spec =
>>>>>>>>> + (next_index ^ next0) | (next_index ^
>>>> next1) |
>>>>>>>>> + (next_index ^ next2) | (next_index ^
>>>> next3);
>>>>>>>>> +
>>>>>>>>> + if (unlikely(fix_spec)) {
>>>>>>>>> + /* Copy things successfully speculated
>>>> till now
>>>>>>>> */
>>>>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>>>>>> sizeof(from[0]));
>>>>>>>>> + from += last_spec;
>>>>>>>>> + to_next += last_spec;
>>>>>>>>> + held += last_spec;
>>>>>>>>> + last_spec = 0;
>>>>>>>>> +
>>>>>>>>> + /* Next0 */
>>>>>>>>> + if (next_index == next0) {
>>>>>>>>> + to_next[0] = from[0];
>>>>>>>>> + to_next++;
>>>>>>>>> + held++;
>>>>>>>>> + } else {
>>>>>>>>> + rte_node_enqueue_x1(graph,
>>>> node,
>>>>>>>> next0,
>>>>>>>>> + from[0]);
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + /* Next1 */
>>>>>>>>> + if (next_index == next1) {
>>>>>>>>> + to_next[0] = from[1];
>>>>>>>>> + to_next++;
>>>>>>>>> + held++;
>>>>>>>>> + } else {
>>>>>>>>> + rte_node_enqueue_x1(graph,
>>>> node,
>>>>>>>> next1,
>>>>>>>>> + from[1]);
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + /* Next2 */
>>>>>>>>> + if (next_index == next2) {
>>>>>>>>> + to_next[0] = from[2];
>>>>>>>>> + to_next++;
>>>>>>>>> + held++;
>>>>>>>>> + } else {
>>>>>>>>> + rte_node_enqueue_x1(graph,
>>>> node,
>>>>>>>> next2,
>>>>>>>>> + from[2]);
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + /* Next3 */
>>>>>>>>> + if (next_index == next3) {
>>>>>>>>> + to_next[0] = from[3];
>>>>>>>>> + to_next++;
>>>>>>>>> + held++;
>>>>>>>>> + } else {
>>>>>>>>> + rte_node_enqueue_x1(graph,
>>>> node,
>>>>>>>> next3,
>>>>>>>>> + from[3]);
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + from += 4;
>>>>>>>>> +
>>>>>>>>> + } else {
>>>>>>>>> + last_spec += 4;
>>>>>>>>> + }
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + while (n_left_from > 0) {
>>>>>>>>> + uint32_t next_hop;
>>>>>>>>> +
>>>>>>>>> + mbuf0 = pkts[0];
>>>>>>>>> +
>>>>>>>>> + pkts += 1;
>>>>>>>>> + n_left_from -= 1;
>>>>>>>>> +
>>>>>>>>> + /* Extract DIP of mbuf0 */
>>>>>>>>> + eth_hdr = rte_pktmbuf_mtod(mbuf0, struct
>>>>>>>> rte_ether_hdr *);
>>>>>>>>> + ipv4_hdr = (struct rte_ipv4_hdr *)(eth_hdr + 1);
>>>>>>>>> + /* Extract cksum, ttl as ipv4 hdr is in cache */
>>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->cksum =
>>>> ipv4_hdr-
>>>>>>>>> hdr_checksum;
>>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr-
>>>>>>>>> time_to_live;
>>>>>>>>> +
>>>>>>>>> + rc = rte_lpm_lookup(lpm,
>>>> rte_be_to_cpu_32(ipv4_hdr-
>>>>>>>>> dst_addr),
>>>>>>>>> + &next_hop);
>>>>>>>>> + next_hop = (rc == 0) ? next_hop : drop_nh;
>>>>>>>>> +
>>>>>>>>> + rte_node_mbuf_priv1(mbuf0)->nh = next_hop
>>>> &
>>>>>>>> 0xFFFF;
>>>>>>>>> + next0 = (next_hop >> 16);
>>>>>>>>> +
>>>>>>>>> + if (unlikely(next_index ^ next0)) {
>>>>>>>>> + /* Copy things successfully speculated
>>>> till now
>>>>>>>> */
>>>>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>>>>>> sizeof(from[0]));
>>>>>>>>> + from += last_spec;
>>>>>>>>> + to_next += last_spec;
>>>>>>>>> + held += last_spec;
>>>>>>>>> + last_spec = 0;
>>>>>>>>> +
>>>>>>>>> + rte_node_enqueue_x1(graph, node,
>>>> next0,
>>>>>>>> from[0]);
>>>>>>>>> + from += 1;
>>>>>>>>> + } else {
>>>>>>>>> + last_spec += 1;
>>>>>>>>> + }
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + /* !!! Home run !!! */
>>>>>>>>> + if (likely(last_spec == nb_objs)) {
>>>>>>>>> + rte_node_next_stream_move(graph, node,
>>>>>>>> next_index);
>>>>>>>>> + return nb_objs;
>>>>>>>>> + }
>>>>>>>>> +
>>>>>>>>> + held += last_spec;
>>>>>>>>> + /* Copy things successfully speculated till now */
>>>>>>>>> + rte_memcpy(to_next, from, last_spec *
>>>> sizeof(from[0]));
>>>>>>>>> + rte_node_next_stream_put(graph, node, next_index,
>>>> held);
>>>>>>>>> +
>>>>>>>>> + return nb_objs;
>>>>>>>>> +}
>>>>>>>>> +
>>>>>>>>> #else
>>>>>>>>>
>>>>>>>>> static uint16_t
>>>>>>>>>
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem
2020-03-18 21:35 [dpdk-dev] [PATCH v1 00/26] graph: introduce graph subsystem jerinj
` (25 preceding siblings ...)
2020-03-18 21:35 ` [dpdk-dev] [PATCH v1 26/26] l3fwd-graph: add graph config and main loop jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 01/28] graph: define the public API for graph support jerinj
` (29 more replies)
26 siblings, 30 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark,
pbhagavatula, ndabilpuram, Jerin Jacob
From: Jerin Jacob <jerinj@marvell.com>
Using graph traversal for packet processing is a proven architecture
that has been implemented in various open source libraries.
Graph architecture for packet processing enables abstracting the data
processing functions as “nodes” and “links” them together to create a
complex “graph” to create reusable/modular data processing functions.
The patchset further includes performance enhancements and modularity
to the DPDK as discussed in more detail below.
v2..v1:
------
1) Added programmer guide/implementation documentation and l3fwd-graph doc
Hosted in netlify for easy reference:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Programmer’s Guide:
https://dpdk-graph.netlify.com/doc/html/guides/prog_guide/graph_lib.html
l3fwd-graph doc:
https://dpdk-graph.netlify.com/doc/html/guides/sample_app_ug/l3_forward_graph.html
API doc:
https://dpdk-graph.netlify.com/doc/html/api/rte__graph_8h.html
https://dpdk-graph.netlify.com/doc/html/api/rte__graph__worker_8h.html
https://dpdk-graph.netlify.com/doc/html/api/rte__node__eth__api_8h.html
https://dpdk-graph.netlify.com/doc/html/api/rte__node__ip4__api_8h.html
2) Added the release notes for the this feature
3) Fix build issues reported by CI for v1:
http://mails.dpdk.org/archives/test-report/2020-March/121326.html
RFC..v1:
--------
1) Split the patch to more logical ones for review.
2) Added doxygen comments for the API
3) Code cleanup
4) Additional performance improvements.
Delta between l3fwd and l3fwd-graph is negligible now.
(~1%) on octeontx2.
5) Added SIMD routines for x86 in additional to arm64.
Addional nodes planned for v20.08
----------------------------------
1) Packet classification node
2) Support for IPV6 LPM node
This patchset contains
-----------------------------
1) The API definition to "create" nodes and "link" together to create a
"graph" for packet processing. See, lib/librte_graph/rte_graph.h
2) The Fast path API definition for the graph walker and enqueue
function used by the workers. See, lib/librte_graph/rte_graph_worker.h
3) Optimized SW implementation for (1) and (2). See, lib/librte_graph/
4) Test case to verify the graph infrastructure functionality
See, app/test/test_graph.c
5) Performance test cases to evaluate the cost of graph walker and nodes
enqueue fast-path function for various combinations.
See app/test/test_graph_perf.c
6) Packet processing nodes(Null, Rx, Tx, Pkt drop, IPV4 rewrite, IPv4
lookup)
using graph infrastructure. See lib/librte_node/*
7) An example application to showcase l3fwd
(functionality same as existing examples/l3fwd) using graph
infrastructure and use packets processing nodes (item (6)). See examples/l3fwd-graph/.
Performance
-----------
1) Graph walk and node enqueue overhead can be tested with performance
test case application [1]
# If all packets go from a node to another node (we call it as
# "homerun") then it will be just a pointer swap for a burst of packets.
# In the worst case, a couple of handful cycles to move an object from a
node to another node.
2) Performance comparison with existing l3fwd (The complete static code
with out any nodes) vs modular l3fwd-graph with 5 nodes
(ip4_lookup, ip4_rewrite, ethdev_tx, ethdev_rx, pkt_drop).
Here is graphical representation of the l3fwd-graph as Graphviz dot
file:
http://bit.ly/39UPPGm
# l3fwd-graph performance is -1.2% wrt static l3fwd.
# We have simulated the similar test with existing librte_pipeline
# application [4].
ip_pipline application is -48.62% wrt static l3fwd.
The above results are on octeontx2. It may vary on other platforms.
The platforms with higher L1 and L2 caches will have further better
performance.
Tested architectures:
--------------------
1) AArch64
2) X86
Identified tweaking for better performance on different targets
---------------------------------------------------------------
1) Test with various burst size values (256, 128, 64, 32) using
CONFIG_RTE_GRAPH_BURST_SIZE config option.
Based on our testing, on x86 and arm64 servers, The sweet spot is 256
burst size.
While on arm64 embedded SoCs, it is either 64 or 128.
2) Disable node statistics (use CONFIG_RTE_LIBRTE_GRAPH_STATS config
option)
if not needed.
3) Use arm64 optimized memory copy for arm64 architecture by
selecting CONFIG_RTE_ARCH_ARM64_MEMCPY.
Commands to run tests
---------------------
[1]
perf test:
echo "graph_perf_autotest" | sudo ./build/app/test/dpdk-test -c 0x30
[2]
functionality test:
echo "graph_autotest" | sudo ./build/app/test/dpdk-test -c 0x30
[3]
l3fwd-graph:
./l3fwd-graph -c 0x100 -- -p 0x3 --config="(0, 0, 8)" -P
[4]
# ./ip_pipeline --c 0xff0000 -- -s route.cli
Route.cli: (Copy paste to the shell to avoid dos format issues)
https://pastebin.com/raw/B4Ktx7TT
Jerin Jacob (13):
graph: define the public API for graph support
graph: implement node registration
graph: implement node operations
graph: implement node debug routines
graph: implement internal graph operation helpers
graph: populate fastpath memory for graph reel
graph: implement create and destroy APIs
graph: implement graph operation APIs
graph: implement Graphviz export
graph: implement debug routines
graph: implement stats support
graph: implement fastpath API routines
doc: add graph library programmer's guide guide
Kiran Kumar K (2):
graph: add unit test case
node: add ipv4 rewrite node
Nithin Dabilpuram (11):
node: add log infra and null node
node: add ethdev Rx node
node: add ethdev Tx node
node: add ethdev Rx and Tx node ctrl API
node: ipv4 lookup for arm64
node: add ipv4 rewrite and lookup ctrl API
node: add pkt drop node
l3fwd-graph: add graph based l3fwd skeleton
l3fwd-graph: add ethdev configuration changes
l3fwd-graph: add graph config and main loop
doc: add l3fwd graph application user guide
Pavan Nikhilesh (2):
graph: add performance testcase
node: ipv4 lookup for x86
MAINTAINERS | 14 +
app/test/Makefile | 7 +
app/test/meson.build | 12 +-
app/test/test_graph.c | 819 ++++
app/test/test_graph_perf.c | 1057 ++++++
config/common_base | 12 +
config/rte_config.h | 4 +
doc/api/doxy-api-index.md | 5 +
doc/api/doxy-api.conf.in | 2 +
doc/guides/prog_guide/graph_lib.rst | 397 ++
.../prog_guide/img/anatomy_of_a_node.svg | 1078 ++++++
.../prog_guide/img/graph_mem_layout.svg | 702 ++++
doc/guides/prog_guide/img/link_the_nodes.svg | 3330 +++++++++++++++++
doc/guides/prog_guide/index.rst | 1 +
doc/guides/rel_notes/release_20_05.rst | 32 +
doc/guides/sample_app_ug/index.rst | 1 +
doc/guides/sample_app_ug/intro.rst | 4 +
doc/guides/sample_app_ug/l3_forward_graph.rst | 327 ++
examples/Makefile | 3 +
examples/l3fwd-graph/Makefile | 58 +
examples/l3fwd-graph/main.c | 1111 ++++++
examples/l3fwd-graph/meson.build | 13 +
examples/meson.build | 6 +-
lib/Makefile | 6 +
lib/librte_graph/Makefile | 28 +
lib/librte_graph/graph.c | 589 +++
lib/librte_graph/graph_debug.c | 84 +
lib/librte_graph/graph_ops.c | 169 +
lib/librte_graph/graph_populate.c | 234 ++
lib/librte_graph/graph_private.h | 347 ++
lib/librte_graph/graph_stats.c | 406 ++
lib/librte_graph/meson.build | 11 +
lib/librte_graph/node.c | 421 +++
lib/librte_graph/rte_graph.h | 786 ++++
lib/librte_graph/rte_graph_version.map | 47 +
lib/librte_graph/rte_graph_worker.h | 542 +++
lib/librte_node/Makefile | 32 +
lib/librte_node/ethdev_ctrl.c | 116 +
lib/librte_node/ethdev_rx.c | 221 ++
lib/librte_node/ethdev_rx_priv.h | 81 +
lib/librte_node/ethdev_tx.c | 86 +
lib/librte_node/ethdev_tx_priv.h | 62 +
lib/librte_node/ip4_lookup.c | 619 +++
lib/librte_node/ip4_rewrite.c | 326 ++
lib/librte_node/ip4_rewrite_priv.h | 77 +
lib/librte_node/log.c | 14 +
lib/librte_node/meson.build | 10 +
lib/librte_node/node_private.h | 96 +
lib/librte_node/null.c | 23 +
lib/librte_node/pkt_drop.c | 26 +
lib/librte_node/rte_node_eth_api.h | 70 +
lib/librte_node/rte_node_ip4_api.h | 87 +
lib/librte_node/rte_node_version.map | 9 +
lib/meson.build | 5 +-
meson.build | 1 +
mk/rte.app.mk | 2 +
56 files changed, 14623 insertions(+), 5 deletions(-)
create mode 100644 app/test/test_graph.c
create mode 100644 app/test/test_graph_perf.c
create mode 100644 doc/guides/prog_guide/graph_lib.rst
create mode 100644 doc/guides/prog_guide/img/anatomy_of_a_node.svg
create mode 100644 doc/guides/prog_guide/img/graph_mem_layout.svg
create mode 100644 doc/guides/prog_guide/img/link_the_nodes.svg
create mode 100644 doc/guides/sample_app_ug/l3_forward_graph.rst
create mode 100644 examples/l3fwd-graph/Makefile
create mode 100644 examples/l3fwd-graph/main.c
create mode 100644 examples/l3fwd-graph/meson.build
create mode 100644 lib/librte_graph/Makefile
create mode 100644 lib/librte_graph/graph.c
create mode 100644 lib/librte_graph/graph_debug.c
create mode 100644 lib/librte_graph/graph_ops.c
create mode 100644 lib/librte_graph/graph_populate.c
create mode 100644 lib/librte_graph/graph_private.h
create mode 100644 lib/librte_graph/graph_stats.c
create mode 100644 lib/librte_graph/meson.build
create mode 100644 lib/librte_graph/node.c
create mode 100644 lib/librte_graph/rte_graph.h
create mode 100644 lib/librte_graph/rte_graph_version.map
create mode 100644 lib/librte_graph/rte_graph_worker.h
create mode 100644 lib/librte_node/Makefile
create mode 100644 lib/librte_node/ethdev_ctrl.c
create mode 100644 lib/librte_node/ethdev_rx.c
create mode 100644 lib/librte_node/ethdev_rx_priv.h
create mode 100644 lib/librte_node/ethdev_tx.c
create mode 100644 lib/librte_node/ethdev_tx_priv.h
create mode 100644 lib/librte_node/ip4_lookup.c
create mode 100644 lib/librte_node/ip4_rewrite.c
create mode 100644 lib/librte_node/ip4_rewrite_priv.h
create mode 100644 lib/librte_node/log.c
create mode 100644 lib/librte_node/meson.build
create mode 100644 lib/librte_node/node_private.h
create mode 100644 lib/librte_node/null.c
create mode 100644 lib/librte_node/pkt_drop.c
create mode 100644 lib/librte_node/rte_node_eth_api.h
create mode 100644 lib/librte_node/rte_node_ip4_api.h
create mode 100644 lib/librte_node/rte_node_version.map
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 01/28] graph: define the public API for graph support
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 02/28] graph: implement node registration jerinj
` (28 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Thomas Monjalon, Bruce Richardson, John McNamara,
Marko Kovacevic, Jerin Jacob, Kiran Kumar K
Cc: dev, david.marchand, mdr, mattias.ronnblom, pbhagavatula, ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Graph architecture abstracts the data processing functions as
"node" and "link" them together to create a complex "graph" to enable
reusable/modular data processing functions.
These APIs enables graph framework operations such as create, lookup,
dump and destroy on graph and node operations such as clone,
edge update, and edge shrink, etc. The API also allows creating the
stats cluster to monitor per graph and per node stats.
This patch defines the public API for graph support.
This patch also adds support for the build infrastructure and
update the MAINTAINERS file for the graph subsystem.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
MAINTAINERS | 5 +
config/common_base | 7 +
config/rte_config.h | 4 +
doc/api/doxy-api-index.md | 1 +
doc/api/doxy-api.conf.in | 1 +
lib/Makefile | 3 +
lib/librte_graph/Makefile | 22 +
lib/librte_graph/graph.c | 5 +
lib/librte_graph/meson.build | 11 +
lib/librte_graph/rte_graph.h | 786 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 3 +
lib/meson.build | 2 +-
mk/rte.app.mk | 1 +
13 files changed, 850 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_graph/Makefile
create mode 100644 lib/librte_graph/graph.c
create mode 100644 lib/librte_graph/meson.build
create mode 100644 lib/librte_graph/rte_graph.h
create mode 100644 lib/librte_graph/rte_graph_version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index db235c2cc..bc7085983 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1469,6 +1469,11 @@ F: examples/bpf/
F: app/test/test_bpf.c
F: doc/guides/prog_guide/bpf_lib.rst
+Graph - EXPERIMENTAL
+M: Jerin Jacob <jerinj@marvell.com>
+M: Kiran Kumar K <kirankumark@marvell.com>
+F: lib/librte_graph/
+
Test Applications
-----------------
diff --git a/config/common_base b/config/common_base
index c31175f9d..32f982136 100644
--- a/config/common_base
+++ b/config/common_base
@@ -1074,6 +1074,13 @@ CONFIG_RTE_LIBRTE_BPF_ELF=n
#
CONFIG_RTE_LIBRTE_IPSEC=y
+#
+# Compile librte_graph
+#
+CONFIG_RTE_LIBRTE_GRAPH=y
+CONFIG_RTE_GRAPH_BURST_SIZE=256
+CONFIG_RTE_LIBRTE_GRAPH_STATS=y
+
#
# Compile the test application
#
diff --git a/config/rte_config.h b/config/rte_config.h
index d30786bc0..e9201fd46 100644
--- a/config/rte_config.h
+++ b/config/rte_config.h
@@ -98,6 +98,10 @@
/* KNI defines */
#define RTE_KNI_PREEMPT_DEFAULT 1
+/* rte_graph defines */
+#define RTE_GRAPH_BURST_SIZE 256
+#define RTE_LIBRTE_GRAPH_STATS 1
+
/****** driver defines ********/
/* QuickAssist device */
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index dff496be0..5cc50f750 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -159,6 +159,7 @@ The public API headers are grouped by topics:
* [pipeline] (@ref rte_pipeline.h)
[port_in_action] (@ref rte_port_in_action.h)
[table_action] (@ref rte_table_action.h)
+ * [graph] (@ref rte_graph.h):
- **basic**:
[approx fraction] (@ref rte_approx.h),
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 1c4392eec..759a7213e 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -33,6 +33,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/lib/librte_eventdev \
@TOPDIR@/lib/librte_fib \
@TOPDIR@/lib/librte_flow_classify \
+ @TOPDIR@/lib/librte_graph \
@TOPDIR@/lib/librte_gro \
@TOPDIR@/lib/librte_gso \
@TOPDIR@/lib/librte_hash \
diff --git a/lib/Makefile b/lib/Makefile
index 46b91ae1a..1f572b659 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -119,6 +119,9 @@ DEPDIRS-librte_telemetry := librte_eal librte_metrics librte_ethdev
DIRS-$(CONFIG_RTE_LIBRTE_RCU) += librte_rcu
DEPDIRS-librte_rcu := librte_eal
+DIRS-$(CONFIG_RTE_LIBRTE_GRAPH) += librte_graph
+DEPDIRS-librte_graph := librte_eal
+
ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
endif
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
new file mode 100644
index 000000000..26fe514f3
--- /dev/null
+++ b/lib/librte_graph/Makefile
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_graph.a
+
+CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+LDLIBS += -lrte_eal
+
+EXPORT_MAP := rte_graph_version.map
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+
+# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
new file mode 100644
index 000000000..a55bf443a
--- /dev/null
+++ b/lib/librte_graph/graph.c
@@ -0,0 +1,5 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include "rte_graph.h"
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
new file mode 100644
index 000000000..455cf2ba5
--- /dev/null
+++ b/lib/librte_graph/meson.build
@@ -0,0 +1,11 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+name = 'graph'
+
+sources = files('graph.c')
+headers = files('rte_graph.h')
+allow_experimental_apis = true
+
+deps += ['eal']
diff --git a/lib/librte_graph/rte_graph.h b/lib/librte_graph/rte_graph.h
new file mode 100644
index 000000000..4bcf0a6e5
--- /dev/null
+++ b/lib/librte_graph/rte_graph.h
@@ -0,0 +1,786 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef _RTE_GRAPH_H_
+#define _RTE_GRAPH_H_
+
+/**
+ * @file rte_graph.h
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Graph architecture abstracts the data processing functions as
+ * "node" and "link" them together to create a complex "graph" to enable
+ * reusable/modular data processing functions.
+ *
+ * This API enables graph framework operations such as create, lookup,
+ * dump and destroy on graph and node operations such as clone,
+ * edge update, and edge shrink, etc. The API also allows to create the stats
+ * cluster to monitor per graph and per node stats.
+ *
+ */
+
+#include <stdbool.h>
+#include <stdio.h>
+
+#include <rte_common.h>
+#include <rte_compat.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define RTE_GRAPH_NAMESIZE 64 /**< Max length of graph name. */
+#define RTE_NODE_NAMESIZE 64 /**< Max length of node name. */
+#define RTE_GRAPH_OFF_INVALID UINT32_MAX /**< Invalid graph offset. */
+#define RTE_NODE_ID_INVALID UINT32_MAX /**< Invalid node id. */
+#define RTE_EDGE_ID_INVALID UINT16_MAX /**< Invalid edge id. */
+#define RTE_GRAPH_ID_INVALID UINT16_MAX /**< Invalid graph id. */
+#define RTE_GRAPH_FENCE 0xdeadbeef12345678ULL /**< Graph fence data. */
+
+typedef uint32_t rte_graph_off_t; /**< Graph offset type. */
+typedef uint32_t rte_node_t; /**< Node id type. */
+typedef uint16_t rte_edge_t; /**< Edge id type. */
+typedef uint16_t rte_graph_t; /**< Graph id type. */
+
+/** Burst size in terms of log2 */
+#if RTE_GRAPH_BURST_SIZE == 1
+#define RTE_GRAPH_BURST_SIZE_LOG2 0 /**< Object burst size of 1. */
+#elif RTE_GRAPH_BURST_SIZE == 2
+#define RTE_GRAPH_BURST_SIZE_LOG2 1 /**< Object burst size of 2. */
+#elif RTE_GRAPH_BURST_SIZE == 4
+#define RTE_GRAPH_BURST_SIZE_LOG2 2 /**< Object burst size of 4. */
+#elif RTE_GRAPH_BURST_SIZE == 8
+#define RTE_GRAPH_BURST_SIZE_LOG2 3 /**< Object burst size of 8. */
+#elif RTE_GRAPH_BURST_SIZE == 16
+#define RTE_GRAPH_BURST_SIZE_LOG2 4 /**< Object burst size of 16. */
+#elif RTE_GRAPH_BURST_SIZE == 32
+#define RTE_GRAPH_BURST_SIZE_LOG2 5 /**< Object burst size of 32. */
+#elif RTE_GRAPH_BURST_SIZE == 64
+#define RTE_GRAPH_BURST_SIZE_LOG2 6 /**< Object burst size of 64. */
+#elif RTE_GRAPH_BURST_SIZE == 128
+#define RTE_GRAPH_BURST_SIZE_LOG2 7 /**< Object burst size of 128. */
+#elif RTE_GRAPH_BURST_SIZE == 256
+#define RTE_GRAPH_BURST_SIZE_LOG2 8 /**< Object burst size of 256. */
+#else
+#error "Unsupported burst size"
+#endif
+
+/* Forward declaration */
+struct rte_node; /**< Node data */
+struct rte_graph; /**< Graph data */
+struct rte_graph_cluster_stats; /**< Stats for Cluster of graphs */
+struct rte_graph_cluster_node_stats; /**< Node stats within cluster of graphs */
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node process function.
+ *
+ * The function invoked when the worker thread walks on nodes using
+ * rte_graph_walk().
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ * @param objs
+ * Pointer to an array of objects to be processed.
+ * @param nb_objs
+ * Number of objects in the array.
+ *
+ * @return
+ * Number of objects processed.
+ *
+ * @see rte_graph_walk()
+ *
+ */
+typedef uint16_t (*rte_node_process_t)(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node initialization function.
+ *
+ * The function invoked when the user creates the graph using rte_graph_create()
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ *
+ * @return
+ * - 0: Success.
+ * -<0: Failure.
+ *
+ * @see rte_graph_create()
+ */
+typedef int (*rte_node_init_t)(const struct rte_graph *graph,
+ struct rte_node *node);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node finalization function.
+ *
+ * The function invoked when the user destroys the graph using
+ * rte_graph_destroy().
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ *
+ * @see rte_graph_destroy()
+ */
+typedef void (*rte_node_fini_t)(const struct rte_graph *graph,
+ struct rte_node *node);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Graph cluster stats callback.
+ *
+ * @param is_first
+ * Flag to denote that stats are of the first node.
+ * @param is_last
+ * Flag to denote that stats are of the last node.
+ * @param cookie
+ * Cookie supplied during stats creation.
+ * @param stats
+ * Node cluster stats data.
+ *
+ * @return
+ * - 0: Success.
+ * -<0: Failure.
+ */
+typedef int (*rte_graph_cluster_stats_cb_t)(bool is_first, bool is_last,
+ void *cookie, const struct rte_graph_cluster_node_stats *stats);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Structure to hold configuration parameters for creating the graph.
+ *
+ * @see rte_graph_create()
+ */
+struct rte_graph_param {
+ int socket_id; /**< Socket id where memory is allocated. */
+ uint16_t nb_node_patterns; /**< Number of node patterns. */
+ const char **node_patterns;
+ /**< Array of node patterns based on shell pattern. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Structure to hold configuration parameters for graph cluster stats create.
+ *
+ * @see rte_graph_cluster_stats_create()
+ */
+struct rte_graph_cluster_stats_param {
+ int socket_id;
+ /**< Socket id where memory is allocated */
+ rte_graph_cluster_stats_cb_t fn;
+ /**< Stats print callback function. NULL value allowed, in that case,
+ * default print stat function used.
+ */
+ RTE_STD_C11
+ union {
+ void *cookie;
+ FILE *f; /**< File pointer to dump the stats when fn == NULL. */
+ };
+ uint16_t nb_graph_patterns; /**< Number of graph patterns. */
+ const char **graph_patterns;
+ /**< Array of graph patterns based on shell pattern. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node cluster stats data structure.
+ *
+ * @see struct rte_graph_cluster_stats_param::fn
+ */
+struct rte_graph_cluster_node_stats {
+ uint64_t ts; /**< Current timestamp. */
+ uint64_t calls; /**< Current number of calls made. */
+ uint64_t objs; /**< Current number of objs processed. */
+ uint64_t cycles; /**< Current number of cycles. */
+
+ uint64_t prev_ts; /**< Previous call timestamp. */
+ uint64_t prev_calls; /**< Previous number of calls. */
+ uint64_t prev_objs; /**< Previous number of processed objs. */
+ uint64_t prev_cycles; /**< Previous number of cycles. */
+
+ uint64_t realloc_count; /**< Realloc count. */
+
+ rte_node_t id; /**< Node identifier of stats. */
+ uint64_t hz; /**< Cycles per seconds. */
+ char name[RTE_NODE_NAMESIZE]; /**< Name of the node. */
+} __rte_cache_aligned;
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Structure defines the node registration parameters.
+ *
+ * @see __rte_node_register(), RTE_NODE_REGISTER()
+ */
+struct rte_node_register {
+ char name[RTE_NODE_NAMESIZE]; /**< Name of the node. */
+ uint64_t flags; /**< Node configuration flag. */
+#define RTE_NODE_SOURCE_F (1ULL << 0) /**< Node type is source. */
+ rte_node_process_t process; /**< Node process function. */
+ rte_node_init_t init; /**< Node init function. */
+ rte_node_fini_t fini; /**< Node fini function. */
+ rte_node_t id; /**< Node Identifier. */
+ rte_node_t parent_id; /**< Identifier of parent node. */
+ rte_edge_t nb_edges; /**< Number of edges from this node. */
+ const char *next_nodes[]; /**< Names of next nodes. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create Graph.
+ *
+ * Create memory reel, detect loops and find isolated nodes.
+ *
+ * @param name
+ * Unique name for this graph.
+ * @param prm
+ * Graph parameter, includes node names and count to be included
+ * in this graph.
+ *
+ * @return
+ * Unique graph id on success, RTE_GRAPH_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_create(const char *name, struct rte_graph_param *prm);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Destroy Graph.
+ *
+ * Free Graph memory reel.
+ *
+ * @param name
+ * Name of the graph to destroy.
+ *
+ * @return
+ * 0 on success, error otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_destroy(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get graph id from graph name.
+ *
+ * @param name
+ * Name of the graph to get id.
+ *
+ * @return
+ * Graph id on success, RTE_GRAPH_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_from_name(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get graph name from graph id.
+ *
+ * @param id
+ * id of the graph to get name.
+ *
+ * @return
+ * Graph name on success, NULL otherwise.
+ */
+__rte_experimental
+char *rte_graph_id_to_name(rte_graph_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Export the graph as graph viz dot file
+ *
+ * @param name
+ * Name of the graph to export.
+ * @param f
+ * File pointer to export the graph.
+ *
+ * @return
+ * 0 on success, error otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_export(const char *name, FILE *f);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get graph object from its name.
+ *
+ * Typical usage of this API to get graph objects in the worker thread and
+ * followed calling rte_graph_walk() in a loop.
+ *
+ * @param name
+ * Name of the graph.
+ *
+ * @return
+ * Graph pointer on success, NULL otherwise.
+ *
+ * @see rte_graph_walk()
+ */
+__rte_experimental
+struct rte_graph *rte_graph_lookup(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get maximum number of graph available.
+ *
+ * @return
+ * Maximum graph count on success, RTE_GRAPH_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_graph_t rte_graph_max_count(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump the graph information to file.
+ *
+ * @param f
+ * File pointer to dump graph info.
+ * @param id
+ * Graph id to get graph info.
+ */
+__rte_experimental
+void rte_graph_dump(FILE *f, rte_graph_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump all graphs information to file
+ *
+ * @param f
+ * File pointer to dump graph info.
+ */
+__rte_experimental
+void rte_graph_list_dump(FILE *f);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump graph information along with node info to file
+ *
+ * @param f
+ * File pointer to dump graph info.
+ * @param graph
+ * Graph pointer to get graph info.
+ * @param all
+ * true to dump nodes in the graph.
+ */
+__rte_experimental
+void rte_graph_obj_dump(FILE *f, struct rte_graph *graph, bool all);
+
+/** Macro to browse rte_node object after the graph creation */
+#define rte_graph_foreach_node(count, off, graph, node) \
+ for (count = 0, off = graph->nodes_start, \
+ node = RTE_PTR_ADD(graph, off); \
+ count < graph->nb_nodes; \
+ off = node->next, node = RTE_PTR_ADD(graph, off), count++)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get node object with in graph from id.
+ *
+ * @param graph_id
+ * Graph id to get node pointer from.
+ * @param node_id
+ * Node id to get node pointer.
+ *
+ * @return
+ * Node pointer on success, NULL otherwise.
+ */
+__rte_experimental
+struct rte_node *rte_graph_node_get(rte_graph_t graph_id, uint32_t node_id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get node pointer with in graph from name.
+ *
+ * @param graph
+ * Graph name to get node pointer from.
+ * @param name
+ * Node name to get the node pointer.
+ *
+ * @return
+ * Node pointer on success, NULL otherwise.
+ */
+__rte_experimental
+struct rte_node *rte_graph_node_get_by_name(const char *graph,
+ const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Create graph stats cluster to aggregate runtime node stats.
+ *
+ * @param prm
+ * Parameters including file pointer to dump stats,
+ * Graph pattern to create cluster and callback function.
+ *
+ * @return
+ * Valid pointer on success, NULL otherwise.
+ */
+__rte_experimental
+struct rte_graph_cluster_stats *rte_graph_cluster_stats_create(
+ const struct rte_graph_cluster_stats_param *prm);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Destroy cluster stats.
+ *
+ * @param stat
+ * Valid cluster pointer to destroy.
+ *
+ */
+__rte_experimental
+void rte_graph_cluster_stats_destroy(struct rte_graph_cluster_stats *stat);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get stats to application.
+ *
+ * @param[out] stat
+ * Cluster status.
+ * @param skip_cb
+ * true to skip callback function invocation.
+ */
+__rte_experimental
+void rte_graph_cluster_stats_get(struct rte_graph_cluster_stats *stat,
+ bool skip_cb);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Reset cluster stats to zero.
+ *
+ * @param stat
+ * Valid cluster stats pointer.
+ */
+__rte_experimental
+void rte_graph_cluster_stats_reset(struct rte_graph_cluster_stats *stat);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Register new packet processing node. Nodes can be registered
+ * dynamically via this call or statically via the RTE_NODE_REGISTER
+ * macro.
+ *
+ * @param node
+ * Valid node pointer with name, process function and next_nodes.
+ *
+ * @return
+ * Valid node id on success, RTE_NODE_ID_INVALID otherwise.
+ *
+ * @see RTE_NODE_REGISTER()
+ */
+__rte_experimental
+rte_node_t __rte_node_register(const struct rte_node_register *node);
+
+/**
+ * Register a static node.
+ *
+ * The static node is registered through the constructor scheme, thereby, it can
+ * be used in a multi-process scenario.
+ *
+ * @param node
+ * Valid node pointer with name, process function, and next_nodes.
+ */
+#define RTE_NODE_REGISTER(node) \
+ RTE_INIT(rte_node_register_##node) \
+ { \
+ node.parent_id = RTE_NODE_ID_INVALID; \
+ node.id = __rte_node_register(&node); \
+ }
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Clone a node from static node(node created from RTE_NODE_REGISTER).
+ *
+ * @param id
+ * Static node id to clone from.
+ * @param name
+ * Name of the new node. The library prepends the parent node name to the
+ * user-specified the name. The final node name will be,
+ * "parent node name" + "-" + name.
+ *
+ * @return
+ * Valid node id on success, RTE_NODE_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_node_t rte_node_clone(rte_node_t id, const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get node id from node name.
+ *
+ * @param name
+ * Valid node name. In the case of the cloned node, the name will be
+ * "parent node name" + "-" + name.
+ *
+ * @return
+ * Valid node id on success, RTE_NODE_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_node_t rte_node_from_name(const char *name);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get node name from node id.
+ *
+ * @param id
+ * Valid node id.
+ *
+ * @return
+ * Valid node name on success, NULL otherwise.
+ */
+__rte_experimental
+char *rte_node_id_to_name(rte_node_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get the number of edges for a node from node id.
+ *
+ * @param id
+ * Valid node id.
+ *
+ * @return
+ * Valid edge count on success, RTE_EDGE_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_edge_t rte_node_edge_count(rte_node_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Update the edges for a node from node id.
+ *
+ * @param id
+ * Valid node id.
+ * @param from
+ * Index to update the edges from. RTE_EDGE_ID_INVALID is valid,
+ * in that case, it will be added to the end of the list.
+ * @param next_nodes
+ * Name of the edges to update.
+ * @param nb_edges
+ * Number of edges to update.
+ *
+ * @return
+ * Valid edge count on success, 0 otherwise.
+ */
+__rte_experimental
+rte_edge_t rte_node_edge_update(rte_node_t id, rte_edge_t from,
+ const char **next_nodes, uint16_t nb_edges);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Shrink the edges to a given size.
+ *
+ * @param id
+ * Valid node id.
+ * @param size
+ * New size to shrink the edges.
+ *
+ * @return
+ * New size on success, RTE_EDGE_ID_INVALID otherwise.
+ */
+__rte_experimental
+rte_edge_t rte_node_edge_shrink(rte_node_t id, rte_edge_t size);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get the edge names from a given node.
+ *
+ * @param id
+ * Valid node id.
+ * @param[out] next_nodes
+ * Buffer to copy the edge names. The NULL value is allowed in that case,
+ * the function returns the size of the array that needs to be allocated.
+ *
+ * @return
+ * When next_nodes == NULL, it returns the size of the array else
+ * number of item copied.
+ */
+__rte_experimental
+rte_node_t rte_node_edge_get(rte_node_t id, char *next_nodes[]);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get maximum nodes created so far.
+ *
+ * @return
+ * Maximum nodes count on success, 0 otherwise.
+ */
+__rte_experimental
+rte_node_t rte_node_max_count(void);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump node info to file.
+ *
+ * @param f
+ * File pointer to dump the node info.
+ * @param id
+ * Node id to get the info.
+ */
+__rte_experimental
+void rte_node_dump(FILE *f, rte_node_t id);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Dump all node info to file.
+ *
+ * @param f
+ * File pointer to dump the node info.
+ */
+__rte_experimental
+void rte_node_list_dump(FILE *f);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Test the validity of node id.
+ *
+ * @param id
+ * Node id to check.
+ *
+ * @return
+ * 1 if valid id, 0 otherwise.
+ */
+static __rte_always_inline int
+rte_node_is_invalid(rte_node_t id)
+{
+ return (id == RTE_NODE_ID_INVALID);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Test the validity of edge id.
+ *
+ * @param id
+ * Edge node id to check.
+ *
+ * @return
+ * 1 if valid id, 0 otherwise.
+ */
+static __rte_always_inline int
+rte_edge_is_invalid(rte_edge_t id)
+{
+ return (id == RTE_EDGE_ID_INVALID);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Test the validity of graph id.
+ *
+ * @param id
+ * Graph id to check.
+ *
+ * @return
+ * 1 if valid id, 0 otherwise.
+ */
+static __rte_always_inline int
+rte_graph_is_invalid(rte_graph_t id)
+{
+ return (id == RTE_GRAPH_ID_INVALID);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Test stats feature support.
+ *
+ * @return
+ * 1 if stats enabled, 0 otherwise.
+ */
+static __rte_always_inline int
+rte_graph_has_stats_feature(void)
+{
+#ifdef RTE_LIBRTE_GRAPH_STATS
+ return RTE_LIBRTE_GRAPH_STATS;
+#else
+ return 0;
+#endif
+}
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_GRAPH_H_ */
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
new file mode 100644
index 000000000..55ef35df5
--- /dev/null
+++ b/lib/librte_graph/rte_graph_version.map
@@ -0,0 +1,3 @@
+EXPERIMENTAL {
+
+};
diff --git a/lib/meson.build b/lib/meson.build
index 9c3cc55d5..c43d86bb9 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -30,7 +30,7 @@ libraries = [
# add pkt framework libs which use other libs from above
'port', 'table', 'pipeline',
# flow_classify lib depends on pkt framework table lib
- 'flow_classify', 'bpf', 'telemetry']
+ 'flow_classify', 'bpf', 'graph', 'telemetry']
if is_windows
libraries = ['kvargs','eal'] # only supported libraries for windows
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index d295ca0a5..b1195f09a 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -98,6 +98,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_CMDLINE) += -lrte_cmdline
_LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
_LDLIBS-$(CONFIG_RTE_LIBRTE_RCU) += -lrte_rcu
+_LDLIBS-$(CONFIG_RTE_LIBRTE_GRAPH) += -lrte_graph
ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_KNI) += -lrte_kni
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 02/28] graph: implement node registration
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 01/28] graph: define the public API for graph support jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 03/28] graph: implement node operations jerinj
` (27 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding rte_node_register() API implementation includes allocating
memory for node object, check for duplicate node name and
add the allocated node to STAILQ node_list for future use.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/Makefile | 1 +
lib/librte_graph/graph.c | 18 +++-
lib/librte_graph/graph_private.h | 75 ++++++++++++++++
lib/librte_graph/meson.build | 2 +-
lib/librte_graph/node.c | 115 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 4 +
6 files changed, 213 insertions(+), 2 deletions(-)
create mode 100644 lib/librte_graph/graph_private.h
create mode 100644 lib/librte_graph/node.c
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 26fe514f3..933d0ee49 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -14,6 +14,7 @@ LDLIBS += -lrte_eal
EXPORT_MAP := rte_graph_version.map
# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
# install header files
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index a55bf443a..a9c124896 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -2,4 +2,20 @@
* Copyright(C) 2020 Marvell International Ltd.
*/
-#include "rte_graph.h"
+#include <rte_spinlock.h>
+
+#include "graph_private.h"
+
+static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
+
+void
+graph_spinlock_lock(void)
+{
+ rte_spinlock_lock(&graph_lock);
+}
+
+void
+graph_spinlock_unlock(void)
+{
+ rte_spinlock_unlock(&graph_lock);
+}
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
new file mode 100644
index 000000000..8b9ff5292
--- /dev/null
+++ b/lib/librte_graph/graph_private.h
@@ -0,0 +1,75 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef _RTE_GRAPH_PRIVATE_H_
+#define _RTE_GRAPH_PRIVATE_H_
+
+#include <inttypes.h>
+#include <sys/queue.h>
+
+#include <rte_common.h>
+#include <rte_eal.h>
+
+#include "rte_graph.h"
+
+/**
+ * @internal
+ *
+ * Structure that holds node internal data.
+ */
+struct node {
+ STAILQ_ENTRY(node) next; /**< Next node in the list. */
+ char name[RTE_NODE_NAMESIZE]; /**< Name of the node. */
+ uint64_t flags; /**< Node configuration flag. */
+ rte_node_process_t process; /**< Node process function. */
+ rte_node_init_t init; /**< Node init function. */
+ rte_node_fini_t fini; /**< Node fini function. */
+ rte_node_t id; /**< Allocated identifier for the node. */
+ rte_node_t parent_id; /**< Parent node identifier. */
+ rte_edge_t nb_edges; /**< Number of edges from this node. */
+ char next_nodes[][RTE_NODE_NAMESIZE]; /**< Names of next nodes. */
+};
+
+/* Node functions */
+STAILQ_HEAD(node_head, node);
+
+/**
+ * @internal
+ *
+ * Get the head of the node list.
+ *
+ * @return
+ * Pointer to the node head.
+ */
+struct node_head *node_list_head_get(void);
+
+/**
+ * @internal
+ *
+ * Get node pointer from node name.
+ *
+ * @param name
+ * Pointer to character string containing the node name.
+ *
+ * @return
+ * Pointer to the node.
+ */
+struct node *node_from_name(const char *name);
+
+/* Lock functions */
+/**
+ * @internal
+ *
+ * Take a lock on the graph internal spin lock.
+ */
+void graph_spinlock_lock(void);
+
+/**
+ * @internal
+ *
+ * Release a lock on the graph internal spin lock.
+ */
+void graph_spinlock_unlock(void);
+
+#endif /* _RTE_GRAPH_PRIVATE_H_ */
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index 455cf2ba5..5754ac23b 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,7 +4,7 @@
name = 'graph'
-sources = files('graph.c')
+sources = files('node.c', 'graph.c')
headers = files('rte_graph.h')
allow_experimental_apis = true
diff --git a/lib/librte_graph/node.c b/lib/librte_graph/node.c
new file mode 100644
index 000000000..7999ca6ed
--- /dev/null
+++ b/lib/librte_graph/node.c
@@ -0,0 +1,115 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <stdbool.h>
+#include <stdio.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_eal.h>
+#include <rte_errno.h>
+#include <rte_string_fns.h>
+
+#include "graph_private.h"
+
+static struct node_head node_list = STAILQ_HEAD_INITIALIZER(node_list);
+static rte_node_t node_id;
+
+#define NODE_ID_CHECK(id) ID_CHECK(id, node_id)
+
+/* Private functions */
+struct node_head *
+node_list_head_get(void)
+{
+ return &node_list;
+}
+
+struct node *
+node_from_name(const char *name)
+{
+ struct node *node;
+
+ STAILQ_FOREACH(node, &node_list, next)
+ if (strncmp(node->name, name, RTE_NODE_NAMESIZE) == 0)
+ return node;
+
+ return NULL;
+}
+
+static bool
+node_has_duplicate_entry(const char *name)
+{
+ struct node *node;
+
+ /* Is duplicate name registered */
+ STAILQ_FOREACH(node, &node_list, next) {
+ if (strncmp(node->name, name, RTE_NODE_NAMESIZE) == 0) {
+ rte_errno = EEXIST;
+ return 1;
+ }
+ }
+ return 0;
+}
+
+/* Public functions */
+rte_node_t
+__rte_node_register(const struct rte_node_register *reg)
+{
+ struct node *node;
+ rte_edge_t i;
+ size_t sz;
+
+ graph_spinlock_lock();
+
+ /* Check sanity */
+ if (reg == NULL || reg->process == NULL) {
+ rte_errno = EINVAL;
+ goto fail;
+ }
+
+ /* Check for duplicate name */
+ if (node_has_duplicate_entry(reg->name))
+ goto fail;
+
+ sz = sizeof(struct node) + (reg->nb_edges * RTE_NODE_NAMESIZE);
+ node = calloc(1, sz);
+ if (node == NULL) {
+ rte_errno = ENOMEM;
+ goto fail;
+ }
+
+ /* Initialize the node */
+ if (rte_strscpy(node->name, reg->name, RTE_NODE_NAMESIZE) < 0) {
+ rte_errno = E2BIG;
+ goto free;
+ }
+ node->flags = reg->flags;
+ node->process = reg->process;
+ node->init = reg->init;
+ node->fini = reg->fini;
+ node->nb_edges = reg->nb_edges;
+ node->parent_id = reg->parent_id;
+ for (i = 0; i < reg->nb_edges; i++) {
+ if (rte_strscpy(node->next_nodes[i], reg->next_nodes[i],
+ RTE_NODE_NAMESIZE) < 0) {
+ rte_errno = E2BIG;
+ goto free;
+ }
+ }
+
+ node->id = node_id++;
+
+ /* Add the node at tail */
+ STAILQ_INSERT_TAIL(&node_list, node, next);
+ graph_spinlock_unlock();
+
+ return node->id;
+free:
+ free(node);
+fail:
+ graph_spinlock_unlock();
+ return RTE_NODE_ID_INVALID;
+}
+
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 55ef35df5..0884c09f1 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -1,3 +1,7 @@
EXPERIMENTAL {
+ global:
+ __rte_node_register;
+
+ local: *;
};
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 03/28] graph: implement node operations
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 01/28] graph: define the public API for graph support jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 02/28] graph: implement node registration jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 04/28] graph: implement node debug routines jerinj
` (26 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding node-specific API implementation like cloning node, updating
edges for the node, shrinking edges of a node, retrieving edges of a
node.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph_private.h | 10 +
lib/librte_graph/node.c | 269 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 10 +
3 files changed, 289 insertions(+)
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index 8b9ff5292..7ed6d01b6 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -13,6 +13,16 @@
#include "rte_graph.h"
+
+#define ID_CHECK(id, id_max) \
+ do { \
+ if ((id) >= (id_max)) { \
+ rte_errno = EINVAL; \
+ goto fail; \
+ } \
+ } while (0)
+
+
/**
* @internal
*
diff --git a/lib/librte_graph/node.c b/lib/librte_graph/node.c
index 7999ca6ed..8de857889 100644
--- a/lib/librte_graph/node.c
+++ b/lib/librte_graph/node.c
@@ -113,3 +113,272 @@ __rte_node_register(const struct rte_node_register *reg)
return RTE_NODE_ID_INVALID;
}
+static int
+clone_name(struct rte_node_register *reg, struct node *node, const char *name)
+{
+ ssize_t sz, rc;
+
+#define SZ RTE_NODE_NAMESIZE
+ rc = rte_strscpy(reg->name, node->name, SZ);
+ if (rc < 0)
+ goto fail;
+ sz = rc;
+ rc = rte_strscpy(reg->name + sz, "-", RTE_MAX((int16_t)(SZ - sz), 0));
+ if (rc < 0)
+ goto fail;
+ sz += rc;
+ sz = rte_strscpy(reg->name + sz, name, RTE_MAX((int16_t)(SZ - sz), 0));
+ if (sz < 0)
+ goto fail;
+
+ return 0;
+fail:
+ rte_errno = E2BIG;
+ return -rte_errno;
+}
+
+static rte_node_t
+node_clone(struct node *node, const char *name)
+{
+ rte_node_t rc = RTE_NODE_ID_INVALID;
+ struct rte_node_register *reg;
+ rte_edge_t i;
+
+ /* Don't allow to clone a node from a cloned node */
+ if (node->parent_id != RTE_NODE_ID_INVALID) {
+ rte_errno = EEXIST;
+ goto fail;
+ }
+
+ /* Check for duplicate name */
+ if (node_has_duplicate_entry(name))
+ goto fail;
+
+ reg = calloc(1, sizeof(*reg) + (sizeof(char *) * node->nb_edges));
+ if (reg == NULL) {
+ rte_errno = ENOMEM;
+ goto fail;
+ }
+
+ /* Clone the source node */
+ reg->flags = node->flags;
+ reg->process = node->process;
+ reg->init = node->init;
+ reg->fini = node->fini;
+ reg->nb_edges = node->nb_edges;
+ reg->parent_id = node->id;
+
+ for (i = 0; i < node->nb_edges; i++)
+ reg->next_nodes[i] = node->next_nodes[i];
+
+ /* Naming ceremony of the new node. name is node->name + "-" + name */
+ if (clone_name(reg, node, name))
+ goto free;
+
+ rc = __rte_node_register(reg);
+free:
+ free(reg);
+fail:
+ return rc;
+}
+
+rte_node_t
+rte_node_clone(rte_node_t id, const char *name)
+{
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ STAILQ_FOREACH(node, &node_list, next)
+ if (node->id == id)
+ return node_clone(node, name);
+
+fail:
+ return RTE_NODE_ID_INVALID;
+}
+
+rte_node_t
+rte_node_from_name(const char *name)
+{
+ struct node *node;
+
+ STAILQ_FOREACH(node, &node_list, next)
+ if (strncmp(node->name, name, RTE_NODE_NAMESIZE) == 0)
+ return node->id;
+
+ return RTE_NODE_ID_INVALID;
+}
+
+char *
+rte_node_id_to_name(rte_node_t id)
+{
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ STAILQ_FOREACH(node, &node_list, next)
+ if (node->id == id)
+ return node->name;
+
+fail:
+ return NULL;
+}
+
+rte_edge_t
+rte_node_edge_count(rte_node_t id)
+{
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ STAILQ_FOREACH(node, &node_list, next)
+ if (node->id == id)
+ return node->nb_edges;
+fail:
+ return RTE_EDGE_ID_INVALID;
+}
+
+static rte_edge_t
+edge_update(struct node *node, struct node *prev, rte_edge_t from,
+ const char **next_nodes, rte_edge_t nb_edges)
+{
+ rte_edge_t i, max_edges, count = 0;
+ struct node *new_node;
+ bool need_realloc;
+ size_t sz;
+
+ if (from == RTE_EDGE_ID_INVALID)
+ from = node->nb_edges;
+
+ /* Don't create hole in next_nodes[] list */
+ if (from > node->nb_edges) {
+ rte_errno = ENOMEM;
+ goto fail;
+ }
+
+ /* Remove me from list */
+ STAILQ_REMOVE(&node_list, node, node, next);
+
+ /* Allocate the storage space for new node if required */
+ max_edges = from + nb_edges;
+ need_realloc = max_edges > node->nb_edges;
+ if (need_realloc) {
+ sz = sizeof(struct node) + (max_edges * RTE_NODE_NAMESIZE);
+ new_node = realloc(node, sz);
+ if (new_node == NULL) {
+ rte_errno = ENOMEM;
+ goto restore;
+ } else {
+ node = new_node;
+ }
+ }
+
+ /* Update the new nodes name */
+ for (i = from; i < max_edges; i++, count++) {
+ if (rte_strscpy(node->next_nodes[i], next_nodes[count],
+ RTE_NODE_NAMESIZE) < 0) {
+ rte_errno = E2BIG;
+ goto restore;
+ }
+ }
+restore:
+ /* Update the linked list to point new node address in prev node */
+ if (prev)
+ STAILQ_INSERT_AFTER(&node_list, prev, node, next);
+ else
+ STAILQ_INSERT_HEAD(&node_list, node, next);
+
+ if (need_realloc)
+ node->nb_edges += count;
+
+fail:
+ return count;
+}
+
+rte_edge_t
+rte_node_edge_shrink(rte_node_t id, rte_edge_t size)
+{
+ rte_edge_t rc = RTE_EDGE_ID_INVALID;
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ graph_spinlock_lock();
+
+ STAILQ_FOREACH(node, &node_list, next) {
+ if (node->id == id) {
+ if (node->nb_edges < size) {
+ rte_errno = E2BIG;
+ goto fail;
+ }
+ node->nb_edges = size;
+ rc = size;
+ break;
+ }
+ }
+
+fail:
+ graph_spinlock_unlock();
+ return rc;
+}
+
+rte_edge_t
+rte_node_edge_update(rte_node_t id, rte_edge_t from, const char **next_nodes,
+ uint16_t nb_edges)
+{
+ rte_edge_t rc = RTE_EDGE_ID_INVALID;
+ struct node *n, *prev;
+
+ NODE_ID_CHECK(id);
+ graph_spinlock_lock();
+
+ prev = NULL;
+ STAILQ_FOREACH(n, &node_list, next) {
+ if (n->id == id) {
+ rc = edge_update(n, prev, from, next_nodes, nb_edges);
+ break;
+ }
+ prev = n;
+ }
+
+ graph_spinlock_unlock();
+fail:
+ return rc;
+}
+
+static rte_node_t
+node_copy_edges(struct node *node, char *next_nodes[])
+{
+ rte_edge_t i;
+
+ for (i = 0; i < node->nb_edges; i++)
+ next_nodes[i] = node->next_nodes[i];
+
+ return i;
+}
+
+rte_node_t
+rte_node_edge_get(rte_node_t id, char *next_nodes[])
+{
+ rte_node_t rc = RTE_NODE_ID_INVALID;
+ struct node *node;
+
+ NODE_ID_CHECK(id);
+ graph_spinlock_lock();
+
+ STAILQ_FOREACH(node, &node_list, next) {
+ if (node->id == id) {
+ if (next_nodes == NULL)
+ rc = sizeof(char *) * node->nb_edges;
+ else
+ rc = node_copy_edges(node, next_nodes);
+ break;
+ }
+ }
+
+ graph_spinlock_unlock();
+fail:
+ return rc;
+}
+
+rte_node_t
+rte_node_max_count(void)
+{
+ return node_id;
+}
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 0884c09f1..412386356 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -3,5 +3,15 @@ EXPERIMENTAL {
__rte_node_register;
+ rte_node_clone;
+ rte_node_edge_count;
+ rte_node_edge_get;
+ rte_node_edge_shrink;
+ rte_node_edge_update;
+ rte_node_from_name;
+ rte_node_id_to_name;
+ rte_node_list_dump;
+ rte_node_max_count;
+
local: *;
};
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 04/28] graph: implement node debug routines
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (2 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 03/28] graph: implement node operations jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 05/28] graph: implement internal graph operation helpers jerinj
` (25 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding node debug API implementation support to dump
single or all the node objects to the given file.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/Makefile | 1 +
lib/librte_graph/graph_debug.c | 25 ++++++++++++++++++++
lib/librte_graph/graph_private.h | 12 ++++++++++
lib/librte_graph/meson.build | 2 +-
lib/librte_graph/node.c | 32 ++++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 1 +
6 files changed, 72 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_graph/graph_debug.c
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 933d0ee49..2a6d86933 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -16,6 +16,7 @@ EXPORT_MAP := rte_graph_version.map
# all source are stored in SRCS-y
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
# install header files
SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
diff --git a/lib/librte_graph/graph_debug.c b/lib/librte_graph/graph_debug.c
new file mode 100644
index 000000000..75238e7ca
--- /dev/null
+++ b/lib/librte_graph/graph_debug.c
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_common.h>
+#include <rte_debug.h>
+
+#include "graph_private.h"
+
+void
+node_dump(FILE *f, struct node *n)
+{
+ rte_edge_t i;
+
+ fprintf(f, "node <%s>\n", n->name);
+ fprintf(f, " id=%" PRIu32 "\n", n->id);
+ fprintf(f, " flags=0x%" PRIx64 "\n", n->flags);
+ fprintf(f, " addr=%p\n", n);
+ fprintf(f, " process=%p\n", n->process);
+ fprintf(f, " nb_edges=%d\n", n->nb_edges);
+
+ for (i = 0; i < n->nb_edges; i++)
+ fprintf(f, " edge[%d] <%s>\n", i, n->next_nodes[i]);
+}
+
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index 7ed6d01b6..6db04cee7 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -82,4 +82,16 @@ void graph_spinlock_lock(void);
*/
void graph_spinlock_unlock(void);
+/**
+ * @internal
+ *
+ * Dump internal node object data.
+ *
+ * @param f
+ * FILE pointer to dump the info.
+ * @param g
+ * Pointer to the internal node object.
+ */
+void node_dump(FILE *f, struct node *n);
+
#endif /* _RTE_GRAPH_PRIVATE_H_ */
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index 5754ac23b..01512182f 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,7 +4,7 @@
name = 'graph'
-sources = files('node.c', 'graph.c')
+sources = files('node.c', 'graph.c', 'graph_debug.c')
headers = files('rte_graph.h')
allow_experimental_apis = true
diff --git a/lib/librte_graph/node.c b/lib/librte_graph/node.c
index 8de857889..2f9c2ea4c 100644
--- a/lib/librte_graph/node.c
+++ b/lib/librte_graph/node.c
@@ -377,6 +377,38 @@ rte_node_edge_get(rte_node_t id, char *next_nodes[])
return rc;
}
+static void
+node_scan_dump(FILE *f, rte_node_t id, bool all)
+{
+ struct node *node;
+
+ RTE_ASSERT(f != NULL);
+ NODE_ID_CHECK(id);
+
+ STAILQ_FOREACH(node, &node_list, next) {
+ if (all == true) {
+ node_dump(f, node);
+ } else if (node->id == id) {
+ node_dump(f, node);
+ return;
+ }
+ }
+fail:
+ return;
+}
+
+void
+rte_node_dump(FILE *f, rte_node_t id)
+{
+ node_scan_dump(f, id, false);
+}
+
+void
+rte_node_list_dump(FILE *f)
+{
+ node_scan_dump(f, 0, true);
+}
+
rte_node_t
rte_node_max_count(void)
{
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 412386356..f2c2139c5 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -4,6 +4,7 @@ EXPERIMENTAL {
__rte_node_register;
rte_node_clone;
+ rte_node_dump;
rte_node_edge_count;
rte_node_edge_get;
rte_node_edge_shrink;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 05/28] graph: implement internal graph operation helpers
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (3 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 04/28] graph: implement node debug routines jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 06/28] graph: populate fastpath memory for graph reel jerinj
` (24 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding internal graph API helpers support to check whether a graph has
isolated nodes and any node have a loop to itself and BFS
algorithm implementation etc.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
---
lib/librte_graph/Makefile | 1 +
lib/librte_graph/graph.c | 2 +-
lib/librte_graph/graph_ops.c | 169 ++++++++++++++++++++++++++++++
lib/librte_graph/graph_private.h | 173 +++++++++++++++++++++++++++++++
lib/librte_graph/meson.build | 2 +-
5 files changed, 345 insertions(+), 2 deletions(-)
create mode 100644 lib/librte_graph/graph_ops.c
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 2a6d86933..39ecb2652 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -16,6 +16,7 @@ EXPORT_MAP := rte_graph_version.map
# all source are stored in SRCS-y
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
# install header files
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index a9c124896..4c3f2fe7b 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -7,7 +7,7 @@
#include "graph_private.h"
static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
-
+int rte_graph_logtype;
void
graph_spinlock_lock(void)
{
diff --git a/lib/librte_graph/graph_ops.c b/lib/librte_graph/graph_ops.c
new file mode 100644
index 000000000..335595311
--- /dev/null
+++ b/lib/librte_graph/graph_ops.c
@@ -0,0 +1,169 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <stdbool.h>
+#include <string.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+
+#include "graph_private.h"
+
+/* Check whether a node has next_node to itself */
+static inline int
+node_has_loop_edge(struct node *node)
+{
+ rte_edge_t i;
+ char *name;
+ int rc = 0;
+
+ for (i = 0; i < node->nb_edges; i++) {
+ if (strncmp(node->name, node->next_nodes[i],
+ RTE_NODE_NAMESIZE) == 0) {
+ name = node->name;
+ rc = 1;
+ SET_ERR_JMP(EINVAL, fail, "Node %s has loop to self",
+ name);
+ }
+ }
+fail:
+ return rc;
+}
+
+int
+graph_node_has_loop_edge(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (node_has_loop_edge(graph_node->node))
+ return 1;
+
+ return 0;
+}
+
+rte_node_t
+graph_src_nodes_count(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ rte_node_t rc = 0;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (graph_node->node->flags & RTE_NODE_SOURCE_F)
+ rc++;
+
+ if (rc == 0)
+ SET_ERR_JMP(EINVAL, fail, "Graph needs at least a source node");
+fail:
+ return rc;
+}
+
+/* Check whether a node has next_node to a source node */
+int
+graph_node_has_edge_to_src_node(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ struct node *node;
+ rte_edge_t i;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ for (i = 0; i < graph_node->node->nb_edges; i++) {
+ node = graph_node->adjacency_list[i]->node;
+ if (node->flags & RTE_NODE_SOURCE_F)
+ SET_ERR_JMP(
+ EEXIST, fail,
+ "Node %s points to the source node %s",
+ graph_node->node->name, node->name);
+ }
+ }
+
+ return 0;
+fail:
+ return 1;
+}
+
+rte_node_t
+graph_nodes_count(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ rte_node_t count = 0;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ count++;
+
+ return count;
+}
+
+void
+graph_mark_nodes_as_not_visited(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ graph_node->visited = false;
+}
+
+int
+graph_bfs(struct graph *graph, struct graph_node *start)
+{
+ struct graph_node **queue, *v, *tmp;
+ uint16_t head = 0, tail = 0;
+ rte_edge_t i;
+ size_t sz;
+
+ sz = sizeof(struct graph_node *) * graph_nodes_count(graph);
+ queue = malloc(sz);
+ if (queue == NULL)
+ SET_ERR_JMP(ENOMEM, fail, "Failed to alloc BFS queue of %zu",
+ sz);
+
+ /* BFS algorithm */
+ queue[tail++] = start;
+ start->visited = true;
+ while (head != tail) {
+ v = queue[head++];
+ for (i = 0; i < v->node->nb_edges; i++) {
+ tmp = v->adjacency_list[i];
+ if (tmp->visited == false) {
+ queue[tail++] = tmp;
+ tmp->visited = true;
+ }
+ }
+ }
+
+ free(queue);
+ return 0;
+
+fail:
+ return -rte_errno;
+}
+
+/* Check whether a node has connected path or parent node */
+int
+graph_has_isolated_node(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ graph_mark_nodes_as_not_visited(graph);
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ if (graph_node->node->flags & RTE_NODE_SOURCE_F) {
+ if (graph_node->node->nb_edges == 0)
+ SET_ERR_JMP(EINVAL, fail,
+ "%s node needs minimum one edge",
+ graph_node->node->name);
+ if (graph_bfs(graph, graph_node))
+ goto fail;
+ }
+ }
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (graph_node->visited == false)
+ SET_ERR_JMP(EINVAL, fail, "Found isolated node %s",
+ graph_node->node->name);
+
+ return 0;
+fail:
+ return 1;
+}
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index 6db04cee7..220a35e2a 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -13,6 +13,16 @@
#include "rte_graph.h"
+extern int rte_graph_logtype;
+
+#define GRAPH_LOG(level, ...) \
+ rte_log(RTE_LOG_##level, rte_graph_logtype, \
+ RTE_FMT("GRAPH: %s():%u " RTE_FMT_HEAD(__VA_ARGS__, ) "\n", \
+ __func__, __LINE__, RTE_FMT_TAIL(__VA_ARGS__, )))
+
+#define graph_err(...) GRAPH_LOG(ERR, __VA_ARGS__)
+#define graph_info(...) GRAPH_LOG(INFO, __VA_ARGS__)
+#define graph_dbg(...) GRAPH_LOG(DEBUG, __VA_ARGS__)
#define ID_CHECK(id, id_max) \
do { \
@@ -22,6 +32,12 @@
} \
} while (0)
+#define SET_ERR_JMP(err, where, fmt, ...) \
+ do { \
+ graph_err(fmt, ##__VA_ARGS__); \
+ rte_errno = err; \
+ goto where; \
+ } while (0)
/**
* @internal
@@ -41,6 +57,52 @@ struct node {
char next_nodes[][RTE_NODE_NAMESIZE]; /**< Names of next nodes. */
};
+/**
+ * @internal
+ *
+ * Structure that holds the graph node data.
+ */
+struct graph_node {
+ STAILQ_ENTRY(graph_node) next; /**< Next graph node in the list. */
+ struct node *node; /**< Pointer to internal node. */
+ bool visited; /**< Flag used in BFS to mark node visited. */
+ struct graph_node *adjacency_list[]; /**< Adjacency list of the node. */
+};
+
+/**
+ * @internal
+ *
+ * Structure that holds graph internal data.
+ */
+struct graph {
+ STAILQ_ENTRY(graph) next;
+ /**< List of graphs. */
+ char name[RTE_GRAPH_NAMESIZE];
+ /**< Name of the graph. */
+ const struct rte_memzone *mz;
+ /**< Memzone to store graph data. */
+ rte_graph_off_t nodes_start;
+ /**< Node memory start offset in graph reel. */
+ rte_node_t src_node_count;
+ /**< Number of source nodes in a graph. */
+ struct rte_graph *graph;
+ /**< Pointer to graph data. */
+ rte_node_t node_count;
+ /**< Total number of nodes. */
+ uint32_t cir_start;
+ /**< Circular buffer start offset in graph reel. */
+ uint32_t cir_mask;
+ /**< Circular buffer mask for wrap around. */
+ rte_graph_t id;
+ /**< Graph identifier. */
+ size_t mem_sz;
+ /**< Memory size of the graph. */
+ int socket;
+ /**< Socket identifier where memory is allocated. */
+ STAILQ_HEAD(gnode_list, graph_node) node_list;
+ /**< Nodes in a graph. */
+};
+
/* Node functions */
STAILQ_HEAD(node_head, node);
@@ -67,6 +129,19 @@ struct node_head *node_list_head_get(void);
*/
struct node *node_from_name(const char *name);
+/* Graph list functions */
+STAILQ_HEAD(graph_head, graph);
+
+/**
+ * @internal
+ *
+ * Get the head of the graph list.
+ *
+ * @return
+ * Pointer to the graph head.
+ */
+struct graph_head *graph_list_head_get(void);
+
/* Lock functions */
/**
* @internal
@@ -82,6 +157,104 @@ void graph_spinlock_lock(void);
*/
void graph_spinlock_unlock(void);
+/* Graph operations */
+/**
+ * @internal
+ *
+ * Run a BFS(Breadth First Search) on the graph marking all
+ * the graph nodes as visited.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ * @param start
+ * Pointer to the starting graph node.
+ *
+ * @return
+ * - 0: Success.
+ * - -ENOMEM: Not enough memory for BFS.
+ */
+int graph_bfs(struct graph *graph, struct graph_node *start);
+
+/**
+ * @internal
+ *
+ * Check if there is an isolated node in the given graph.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: No isolated node found.
+ * - 1: Isolated node found.
+ */
+int graph_has_isolated_node(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Check whether a node in the graph has next_node to a source node.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: Node has an edge to source node.
+ * - 1: Node doesn't have an edge to source node.
+ */
+int graph_node_has_edge_to_src_node(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Checks whether node in the graph has a edge to itself i.e. forms a
+ * loop.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: Node has an edge to itself.
+ * - 1: Node doesn't have an edge to itself.
+ */
+int graph_node_has_loop_edge(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Get the count of source nodes in the graph.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * Number of source nodes.
+ */
+rte_node_t graph_src_nodes_count(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Get the count of total number of nodes in the graph.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * Number of nodes.
+ */
+rte_node_t graph_nodes_count(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Clear the visited flag of all the nodes in the graph.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ */
+void graph_mark_nodes_as_not_visited(struct graph *graph);
+
+
/**
* @internal
*
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index 01512182f..16e0625c1 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,7 +4,7 @@
name = 'graph'
-sources = files('node.c', 'graph.c', 'graph_debug.c')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c')
headers = files('rte_graph.h')
allow_experimental_apis = true
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 06/28] graph: populate fastpath memory for graph reel
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (4 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 05/28] graph: implement internal graph operation helpers jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 07/28] graph: implement create and destroy APIs jerinj
` (23 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding support to create and populate the memory for graph reel.
This includes reserving the memory in the memzone, populating the nodes,
Allocating memory for node-specific streams to hold objects.
Once it is populated the reel memory contains the following sections.
+---------------------+
| Graph Header |
+---------------------+
| Fence |
+---------------------+
| Circular buffer |
+---------------------+
| Fence |
+---------------------+
| Node Object 0 |
+------------------- -+
| Node Object 1 |
+------------------- -+
| Node Object 2 |
+------------------- -+
| Node Object n |
+------------------- -+
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/Makefile | 2 +
lib/librte_graph/graph.c | 16 ++
lib/librte_graph/graph_populate.c | 234 +++++++++++++++++++++++++
lib/librte_graph/graph_private.h | 64 +++++++
lib/librte_graph/meson.build | 4 +-
lib/librte_graph/node.c | 5 +
lib/librte_graph/rte_graph_version.map | 1 +
lib/librte_graph/rte_graph_worker.h | 108 ++++++++++++
8 files changed, 432 insertions(+), 2 deletions(-)
create mode 100644 lib/librte_graph/graph_populate.c
create mode 100644 lib/librte_graph/rte_graph_worker.h
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 39ecb2652..7bfd7d51f 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -18,8 +18,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_populate.c
# install header files
SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph.h
+SYMLINK-$(CONFIG_RTE_LIBRTE_GRAPH)-include += rte_graph_worker.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index 4c3f2fe7b..e1930b7d2 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -2,6 +2,7 @@
* Copyright(C) 2020 Marvell International Ltd.
*/
+#include <rte_malloc.h>
#include <rte_spinlock.h>
#include "graph_private.h"
@@ -19,3 +20,18 @@ graph_spinlock_unlock(void)
{
rte_spinlock_unlock(&graph_lock);
}
+
+void __rte_noinline
+__rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
+{
+ uint16_t size = node->size;
+
+ RTE_VERIFY(size != UINT16_MAX);
+ /* Allocate double amount of size to avoid immediate realloc */
+ size = RTE_MIN(UINT16_MAX, RTE_MAX(RTE_GRAPH_BURST_SIZE, size * 2));
+ node->objs = rte_realloc_socket(node->objs, size * sizeof(void *),
+ RTE_CACHE_LINE_SIZE, graph->socket);
+ RTE_VERIFY(node->objs);
+ node->size = size;
+ node->realloc_count++;
+}
diff --git a/lib/librte_graph/graph_populate.c b/lib/librte_graph/graph_populate.c
new file mode 100644
index 000000000..093512efa
--- /dev/null
+++ b/lib/librte_graph/graph_populate.c
@@ -0,0 +1,234 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <fnmatch.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
+
+#include "graph_private.h"
+
+static size_t
+graph_fp_mem_calc_size(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ rte_node_t val;
+ size_t sz;
+
+ /* Graph header */
+ sz = sizeof(struct rte_graph);
+ /* Source nodes list */
+ sz += sizeof(rte_graph_off_t) * graph->src_node_count;
+ /* Circular buffer for pending streams of size number of nodes */
+ val = rte_align32pow2(graph->node_count * sizeof(rte_graph_off_t));
+ sz = RTE_ALIGN(sz, val);
+ graph->cir_start = sz;
+ graph->cir_mask = rte_align32pow2(graph->node_count) - 1;
+ sz += val;
+ /* Fence */
+ sz += sizeof(RTE_GRAPH_FENCE);
+ sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
+ graph->nodes_start = sz;
+ /* For 0..N node objects with fence */
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ sz = RTE_ALIGN(sz, RTE_CACHE_LINE_SIZE);
+ sz += sizeof(struct rte_node);
+ /* Pointer to next nodes(edges) */
+ sz += sizeof(struct rte_node *) * graph_node->node->nb_edges;
+ }
+
+ graph->mem_sz = sz;
+ return sz;
+}
+
+static void
+graph_header_popluate(struct graph *_graph)
+{
+ struct rte_graph *graph = _graph->graph;
+
+ graph->tail = 0;
+ graph->head = (int32_t)-_graph->src_node_count;
+ graph->cir_mask = _graph->cir_mask;
+ graph->nb_nodes = _graph->node_count;
+ graph->cir_start = RTE_PTR_ADD(graph, _graph->cir_start);
+ graph->nodes_start = _graph->nodes_start;
+ graph->socket = _graph->socket;
+ graph->id = _graph->id;
+ memcpy(graph->name, _graph->name, RTE_GRAPH_NAMESIZE);
+ graph->fence = RTE_GRAPH_FENCE;
+}
+
+static void
+graph_nodes_populate(struct graph *_graph)
+{
+ rte_graph_off_t off = _graph->nodes_start;
+ struct rte_graph *graph = _graph->graph;
+ struct graph_node *graph_node;
+ rte_edge_t count, nb_edges;
+ const char *parent;
+ rte_node_t pid;
+
+ STAILQ_FOREACH(graph_node, &_graph->node_list, next) {
+ struct rte_node *node = RTE_PTR_ADD(graph, off);
+ memset(node, 0, sizeof(*node));
+ node->fence = RTE_GRAPH_FENCE;
+ node->off = off;
+ node->process = graph_node->node->process;
+ memcpy(node->name, graph_node->node->name, RTE_GRAPH_NAMESIZE);
+ pid = graph_node->node->parent_id;
+ if (pid != RTE_NODE_ID_INVALID) { /* Cloned node */
+ parent = rte_node_id_to_name(pid);
+ memcpy(node->parent, parent, RTE_GRAPH_NAMESIZE);
+ }
+ node->id = graph_node->node->id;
+ node->parent_id = pid;
+ nb_edges = graph_node->node->nb_edges;
+ node->nb_edges = nb_edges;
+ off += sizeof(struct rte_node);
+ /* Copy the name in first pass to replace with rte_node* later*/
+ for (count = 0; count < nb_edges; count++)
+ node->nodes[count] = (struct rte_node *)&graph_node
+ ->adjacency_list[count]
+ ->node->name[0];
+
+ off += sizeof(struct rte_node *) * nb_edges;
+ off = RTE_ALIGN(off, RTE_CACHE_LINE_SIZE);
+ node->next = off;
+ __rte_node_stream_alloc(graph, node);
+ }
+}
+
+struct rte_node *
+graph_node_id_to_ptr(const struct rte_graph *graph, rte_node_t id)
+{
+ rte_node_t count;
+ rte_graph_off_t off;
+ struct rte_node *node;
+
+ rte_graph_foreach_node(count, off, graph, node)
+ if (unlikely(node->id == id))
+ return node;
+
+ return NULL;
+}
+
+struct rte_node *
+graph_node_name_to_ptr(const struct rte_graph *graph, const char *name)
+{
+ rte_node_t count;
+ rte_graph_off_t off;
+ struct rte_node *node;
+
+ rte_graph_foreach_node(count, off, graph, node)
+ if (strncmp(name, node->name, RTE_NODE_NAMESIZE) == 0)
+ return node;
+
+ return NULL;
+}
+
+static int
+graph_node_nexts_populate(struct graph *_graph)
+{
+ rte_node_t count, val;
+ rte_graph_off_t off;
+ struct rte_node *node;
+ const struct rte_graph *graph = _graph->graph;
+ const char *name;
+
+ rte_graph_foreach_node(count, off, graph, node) {
+ for (val = 0; val < node->nb_edges; val++) {
+ name = (const char *)node->nodes[val];
+ node->nodes[val] = graph_node_name_to_ptr(graph, name);
+ if (node->nodes[val] == NULL)
+ SET_ERR_JMP(EINVAL, fail, "%s not found", name);
+ }
+ }
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static int
+graph_src_nodes_populate(struct graph *_graph)
+{
+ struct rte_graph *graph = _graph->graph;
+ struct graph_node *graph_node;
+ struct rte_node *node;
+ int32_t head = -1;
+ const char *name;
+
+ STAILQ_FOREACH(graph_node, &_graph->node_list, next) {
+ if (graph_node->node->flags & RTE_NODE_SOURCE_F) {
+ name = graph_node->node->name;
+ node = graph_node_name_to_ptr(graph, name);
+ if (node == NULL)
+ SET_ERR_JMP(EINVAL, fail, "%s not found", name);
+
+ __rte_node_stream_alloc(graph, node);
+ graph->cir_start[head--] = node->off;
+ }
+ }
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static int
+graph_fp_mem_populate(struct graph *graph)
+{
+ int rc;
+
+ graph_header_popluate(graph);
+ graph_nodes_populate(graph);
+ rc = graph_node_nexts_populate(graph);
+ rc |= graph_src_nodes_populate(graph);
+
+ return rc;
+}
+
+int
+graph_fp_mem_create(struct graph *graph)
+{
+ const struct rte_memzone *mz;
+ size_t sz;
+
+ sz = graph_fp_mem_calc_size(graph);
+ mz = rte_memzone_reserve(graph->name, sz, graph->socket, 0);
+ if (mz == NULL)
+ SET_ERR_JMP(ENOMEM, fail, "Memzone %s reserve failed",
+ graph->name);
+
+ graph->graph = mz->addr;
+ graph->mz = mz;
+
+ return graph_fp_mem_populate(graph);
+fail:
+ return -rte_errno;
+}
+
+static void
+graph_nodes_mem_destroy(struct rte_graph *graph)
+{
+ rte_node_t count;
+ rte_graph_off_t off;
+ struct rte_node *node;
+
+ if (graph == NULL)
+ return;
+
+ rte_graph_foreach_node(count, off, graph, node)
+ rte_free(node->objs);
+}
+
+int
+graph_fp_mem_destroy(struct graph *graph)
+{
+ graph_nodes_mem_destroy(graph->graph);
+ return rte_memzone_free(graph->mz);
+}
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index 220a35e2a..7fce52e00 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -12,6 +12,7 @@
#include <rte_eal.h>
#include "rte_graph.h"
+#include "rte_graph_worker.h"
extern int rte_graph_logtype;
@@ -254,6 +255,69 @@ rte_node_t graph_nodes_count(struct graph *graph);
*/
void graph_mark_nodes_as_not_visited(struct graph *graph);
+/* Fast path graph memory populate unctions */
+
+/**
+ * @internal
+ *
+ * Create fast-path memory for the graph and nodes.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: Success.
+ * - -ENOMEM: Not enough for graph and nodes.
+ * - -EINVAL: Graph nodes not found.
+ */
+int graph_fp_mem_create(struct graph *graph);
+
+/**
+ * @internal
+ *
+ * Free fast-path memory used by graph and nodes.
+ *
+ * @param graph
+ * Pointer to the internal graph object.
+ *
+ * @return
+ * - 0: Success.
+ * - <0: Graph memzone related error.
+ */
+int graph_fp_mem_destroy(struct graph *graph);
+
+/* Lookup functions */
+/**
+ * @internal
+ *
+ * Get graph node object from node id.
+ *
+ * @param graph
+ * Pointer to rte_graph object.
+ * @param id
+ * Node Identifier.
+ *
+ * @return
+ * Pointer to rte_node if identifier is valid else NULL.
+ */
+struct rte_node *graph_node_id_to_ptr(const struct rte_graph *graph,
+ rte_node_t id);
+
+/**
+ * @internal
+ *
+ * Get graph node object from node name.
+ *
+ * @param graph
+ * Pointer to rte_graph object.
+ * @param node_name
+ * Pointer to character string holding the node name.
+ *
+ * @return
+ * Pointer to rte_node if identifier is valid else NULL.
+ */
+struct rte_node *graph_node_name_to_ptr(const struct rte_graph *graph,
+ const char *node_name);
/**
* @internal
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index 16e0625c1..fb203a5e2 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,8 +4,8 @@
name = 'graph'
-sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c')
-headers = files('rte_graph.h')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_populate.c')
+headers = files('rte_graph.h', 'rte_graph_worker.h')
allow_experimental_apis = true
deps += ['eal']
diff --git a/lib/librte_graph/node.c b/lib/librte_graph/node.c
index 2f9c2ea4c..639269870 100644
--- a/lib/librte_graph/node.c
+++ b/lib/librte_graph/node.c
@@ -61,6 +61,11 @@ __rte_node_register(const struct rte_node_register *reg)
rte_edge_t i;
size_t sz;
+ /* Limit Node specific metadata to one cacheline on 64B CL machine */
+ RTE_BUILD_BUG_ON((offsetof(struct rte_node, nodes) -
+ offsetof(struct rte_node, ctx)) !=
+ RTE_CACHE_LINE_MIN_SIZE);
+
graph_spinlock_lock();
/* Check sanity */
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index f2c2139c5..a9fe1b610 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -2,6 +2,7 @@ EXPERIMENTAL {
global:
__rte_node_register;
+ __rte_node_stream_alloc;
rte_node_clone;
rte_node_dump;
diff --git a/lib/librte_graph/rte_graph_worker.h b/lib/librte_graph/rte_graph_worker.h
new file mode 100644
index 000000000..a8133739d
--- /dev/null
+++ b/lib/librte_graph/rte_graph_worker.h
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef _RTE_GRAPH_WORKER_H_
+#define _RTE_GRAPH_WORKER_H_
+
+/**
+ * @file rte_graph_worker.h
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * This API allows a worker thread to walk over a graph and nodes to create,
+ * process, enqueue and move streams of objects to the next nodes.
+ */
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_prefetch.h>
+#include <rte_memcpy.h>
+#include <rte_memory.h>
+
+#include "rte_graph.h"
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/**
+ * @internal
+ *
+ * Data structure to hold graph data.
+ */
+struct rte_graph {
+ uint32_t tail; /**< Tail of circular buffer. */
+ uint32_t head; /**< Head of circular buffer. */
+ uint32_t cir_mask; /**< Circular buffer wrap around mask. */
+ rte_node_t nb_nodes; /**< Number of nodes in the graph. */
+ rte_graph_off_t *cir_start; /**< Pointer to circular buffer. */
+ rte_graph_off_t nodes_start; /**< Offset at which node memory starts. */
+ rte_graph_t id; /**< Graph identifier. */
+ int socket; /**< Socket ID where memory is allocated. */
+ char name[RTE_GRAPH_NAMESIZE]; /**< Name of the graph. */
+ uint64_t fence; /**< Fence. */
+} __rte_cache_aligned;
+
+/**
+ * @internal
+ *
+ * Data structure to hold node data.
+ */
+struct rte_node {
+ /* Slow path area */
+ uint64_t fence; /**< Fence. */
+ rte_graph_off_t next; /**< Index to next node. */
+ rte_node_t id; /**< Node identifier. */
+ rte_node_t parent_id; /**< Parent Node identifier. */
+ rte_edge_t nb_edges; /**< Number of edges from this node. */
+ uint32_t realloc_count; /**< Number of times realloced. */
+
+ char parent[RTE_NODE_NAMESIZE]; /**< Parent node name. */
+ char name[RTE_NODE_NAMESIZE]; /**< Name of the node. */
+
+ /* Fast path area */
+#define RTE_NODE_CTX_SZ 16
+ uint8_t ctx[RTE_NODE_CTX_SZ] __rte_cache_aligned; /**< Node Context. */
+ uint16_t size; /**< Total number of objects available. */
+ uint16_t idx; /**< Number of objects used. */
+ rte_graph_off_t off; /**< Offset of node in the graph reel. */
+ uint64_t total_cycles; /**< Cycles spent in this node. */
+ uint64_t total_calls; /**< Calls done to this node. */
+ uint64_t total_objs; /**< Objects processed by this node. */
+ RTE_STD_C11
+ union {
+ void **objs; /**< Array of object pointers. */
+ uint64_t objs_u64;
+ };
+ RTE_STD_C11
+ union {
+ rte_node_process_t process; /**< Process function. */
+ uint64_t process_u64;
+ };
+ struct rte_node *nodes[] __rte_cache_min_aligned; /**< Next nodes. */
+} __rte_cache_aligned;
+
+/**
+ * @internal
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Allocate a stream of objects.
+ *
+ * If stream already exists then re-allocate it to a larger size.
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ */
+__rte_experimental
+void __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* _RTE_GRAPH_WORKER_H_ */
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 07/28] graph: implement create and destroy APIs
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (5 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 06/28] graph: populate fastpath memory for graph reel jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 08/28] graph: implement graph operation APIs jerinj
` (22 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding graph specific API implementations like graph create
and graph destroy. This detect loops in the graph,
check for isolated nodes and operation to verify the validity of
graph.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph.c | 320 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 2 +
2 files changed, 322 insertions(+)
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index e1930b7d2..dc373231e 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -2,13 +2,33 @@
* Copyright(C) 2020 Marvell International Ltd.
*/
+#include <fnmatch.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_debug.h>
+#include <rte_errno.h>
#include <rte_malloc.h>
+#include <rte_memzone.h>
#include <rte_spinlock.h>
+#include <rte_string_fns.h>
#include "graph_private.h"
+static struct graph_head graph_list = STAILQ_HEAD_INITIALIZER(graph_list);
static rte_spinlock_t graph_lock = RTE_SPINLOCK_INITIALIZER;
+static rte_graph_t graph_id;
int rte_graph_logtype;
+
+#define GRAPH_ID_CHECK(id) ID_CHECK(id, graph_id)
+
+/* Private functions */
+struct graph_head *
+graph_list_head_get(void)
+{
+ return &graph_list;
+}
+
void
graph_spinlock_lock(void)
{
@@ -21,6 +41,306 @@ graph_spinlock_unlock(void)
rte_spinlock_unlock(&graph_lock);
}
+static int
+graph_node_add(struct graph *graph, struct node *node)
+{
+ struct graph_node *graph_node;
+ size_t sz;
+
+ /* Skip the duplicate nodes */
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (strncmp(node->name, graph_node->node->name,
+ RTE_NODE_NAMESIZE) == 0)
+ return 0;
+
+ /* Allocate new graph node object */
+ sz = sizeof(*graph_node) + node->nb_edges * sizeof(struct node *);
+ graph_node = calloc(1, sz);
+
+ if (graph_node == NULL)
+ SET_ERR_JMP(ENOMEM, free, "Failed to calloc %s object",
+ node->name);
+
+ /* Initialize the graph node */
+ graph_node->node = node;
+
+ /* Add to graph node list */
+ STAILQ_INSERT_TAIL(&graph->node_list, graph_node, next);
+ return 0;
+
+free:
+ free(graph_node);
+ return -rte_errno;
+}
+
+static struct graph_node *
+node_to_graph_node(struct graph *graph, struct node *node)
+{
+ struct graph_node *graph_node;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (graph_node->node == node)
+ return graph_node;
+
+ SET_ERR_JMP(ENODEV, fail, "Found isolated node %s", node->name);
+fail:
+ return NULL;
+}
+
+static int
+graph_node_edges_add(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ struct node *adjacency;
+ const char *next;
+ rte_edge_t i;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ for (i = 0; i < graph_node->node->nb_edges; i++) {
+ next = graph_node->node->next_nodes[i];
+ adjacency = node_from_name(next);
+ if (adjacency == NULL)
+ SET_ERR_JMP(EINVAL, fail,
+ "Node %s not registered", next);
+ if (graph_node_add(graph, adjacency))
+ goto fail;
+ }
+ }
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static int
+graph_adjacency_list_update(struct graph *graph)
+{
+ struct graph_node *graph_node, *tmp;
+ struct node *adjacency;
+ const char *next;
+ rte_edge_t i;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ for (i = 0; i < graph_node->node->nb_edges; i++) {
+ next = graph_node->node->next_nodes[i];
+ adjacency = node_from_name(next);
+ if (adjacency == NULL)
+ SET_ERR_JMP(EINVAL, fail,
+ "Node %s not registered", next);
+ tmp = node_to_graph_node(graph, adjacency);
+ if (tmp == NULL)
+ goto fail;
+ graph_node->adjacency_list[i] = tmp;
+ }
+ }
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static int
+expand_pattern_to_node(struct graph *graph, const char *pattern)
+{
+ struct node_head *node_head = node_list_head_get();
+ bool found = false;
+ struct node *node;
+
+ /* Check for pattern match */
+ STAILQ_FOREACH(node, node_head, next) {
+ if (fnmatch(pattern, node->name, 0) == 0) {
+ if (graph_node_add(graph, node))
+ goto fail;
+ found = true;
+ }
+ }
+ if (found == false)
+ SET_ERR_JMP(EFAULT, fail, "Pattern %s node not found", pattern);
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+static void
+graph_cleanup(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ while (!STAILQ_EMPTY(&graph->node_list)) {
+ graph_node = STAILQ_FIRST(&graph->node_list);
+ STAILQ_REMOVE_HEAD(&graph->node_list, next);
+ free(graph_node);
+ }
+}
+
+static int
+graph_node_init(struct graph *graph)
+{
+ struct graph_node *graph_node;
+ const char *name;
+ int rc;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ if (graph_node->node->init) {
+ name = graph_node->node->name;
+ rc = graph_node->node->init(
+ graph->graph,
+ graph_node_name_to_ptr(graph->graph, name));
+ if (rc)
+ SET_ERR_JMP(rc, err, "Node %s init() failed",
+ name);
+ }
+ }
+
+ return 0;
+err:
+ return -rte_errno;
+}
+
+static void
+graph_node_fini(struct graph *graph)
+{
+ struct graph_node *graph_node;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next)
+ if (graph_node->node->fini)
+ graph_node->node->fini(
+ graph->graph,
+ graph_node_name_to_ptr(graph->graph,
+ graph_node->node->name));
+}
+
+rte_graph_t
+rte_graph_create(const char *name, struct rte_graph_param *prm)
+{
+ struct graph *graph;
+ const char *pattern;
+ uint16_t i;
+
+ graph_spinlock_lock();
+
+ /* Check arguments sanity */
+ if (prm == NULL)
+ SET_ERR_JMP(EINVAL, fail, "Param should not be NULL");
+
+ if (name == NULL)
+ SET_ERR_JMP(EINVAL, fail, "Graph name should not be NULL");
+
+ /* Check for existence of duplicate graph */
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (strncmp(name, graph->name, RTE_GRAPH_NAMESIZE) == 0)
+ SET_ERR_JMP(EEXIST, fail, "Found duplicate graph %s",
+ name);
+
+ /* Create graph object */
+ graph = calloc(1, sizeof(*graph));
+ if (graph == NULL)
+ SET_ERR_JMP(ENOMEM, fail, "Failed to calloc graph object");
+
+ /* Initialize the graph object */
+ STAILQ_INIT(&graph->node_list);
+ if (rte_strscpy(graph->name, name, RTE_GRAPH_NAMESIZE) < 0)
+ SET_ERR_JMP(E2BIG, free, "Too big name=%s", name);
+
+ /* Expand node pattern and add the nodes to the graph */
+ for (i = 0; i < prm->nb_node_patterns; i++) {
+ pattern = prm->node_patterns[i];
+ if (expand_pattern_to_node(graph, pattern))
+ goto graph_cleanup;
+ }
+
+ /* Go over all the nodes edges and add them to the graph */
+ if (graph_node_edges_add(graph))
+ goto graph_cleanup;
+
+ /* Update adjacency list of all nodes in the graph */
+ if (graph_adjacency_list_update(graph))
+ goto graph_cleanup;
+
+ /* Make sure at least a source node present in the graph */
+ if (!graph_src_nodes_count(graph))
+ goto graph_cleanup;
+
+ /* Make sure no node is pointing to source node */
+ if (graph_node_has_edge_to_src_node(graph))
+ goto graph_cleanup;
+
+ /* Don't allow node has loop to self */
+ if (graph_node_has_loop_edge(graph))
+ goto graph_cleanup;
+
+ /* Do BFS from src nodes on the graph to find isolated nodes */
+ if (graph_has_isolated_node(graph))
+ goto graph_cleanup;
+
+ /* Initialize graph object */
+ graph->socket = prm->socket_id;
+ graph->src_node_count = graph_src_nodes_count(graph);
+ graph->node_count = graph_nodes_count(graph);
+ graph->id = graph_id;
+
+ /* Allocate the Graph fast path memory and populate the data */
+ if (graph_fp_mem_create(graph))
+ goto graph_cleanup;
+
+ /* Call init() of the all the nodes in the graph */
+ if (graph_node_init(graph))
+ goto graph_mem_destroy;
+
+ /* All good, Lets add the graph to the list */
+ graph_id++;
+ STAILQ_INSERT_TAIL(&graph_list, graph, next);
+
+ graph_spinlock_unlock();
+ return graph->id;
+
+graph_mem_destroy:
+ graph_fp_mem_destroy(graph);
+graph_cleanup:
+ graph_cleanup(graph);
+free:
+ free(graph);
+fail:
+ graph_spinlock_unlock();
+ return RTE_GRAPH_ID_INVALID;
+}
+
+rte_graph_t
+rte_graph_destroy(const char *graph_name)
+{
+ rte_graph_t rc = RTE_GRAPH_ID_INVALID;
+ struct graph *graph, *tmp;
+ const char *name;
+
+ graph_spinlock_lock();
+
+ graph = STAILQ_FIRST(&graph_list);
+ while (graph != NULL) {
+ tmp = STAILQ_NEXT(graph, next);
+ name = graph->name;
+ if (strncmp(name, graph_name, RTE_GRAPH_NAMESIZE) == 0) {
+ /* Call fini() of the all the nodes in the graph */
+ graph_node_fini(graph);
+ /* Destroy graph fast path memory */
+ rc = graph_fp_mem_destroy(graph);
+ if (rc)
+ SET_ERR_JMP(rc, done, "MZ %s free failed",
+ name);
+
+ graph_cleanup(graph);
+ STAILQ_REMOVE(&graph_list, graph, graph, next);
+ rc = graph->id;
+ free(graph);
+ graph_id--;
+ goto done;
+ }
+ graph = tmp;
+ }
+done:
+ graph_spinlock_unlock();
+ return rc;
+}
+
void __rte_noinline
__rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
{
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index a9fe1b610..dcbd78c02 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -4,6 +4,8 @@ EXPERIMENTAL {
__rte_node_register;
__rte_node_stream_alloc;
+ rte_graph_create;
+ rte_graph_destroy;
rte_node_clone;
rte_node_dump;
rte_node_edge_count;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 08/28] graph: implement graph operation APIs
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (6 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 07/28] graph: implement create and destroy APIs jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 09/28] graph: implement Graphviz export jerinj
` (21 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K, Anatoly Burakov
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding support for graph specific API implementation like
Graph lookup to get graph object, retrieving graph ID
From name and graph name from ID.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph.c | 131 +++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 8 ++
2 files changed, 139 insertions(+)
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index dc373231e..7c6a7897d 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -210,6 +210,54 @@ graph_node_fini(struct graph *graph)
graph_node->node->name));
}
+static struct rte_graph *
+graph_mem_fixup_node_ctx(struct rte_graph *graph)
+{
+ struct rte_node *node;
+ struct node *node_db;
+ rte_graph_off_t off;
+ rte_node_t count;
+ const char *name;
+
+ rte_graph_foreach_node(count, off, graph, node) {
+ if (node->parent_id == RTE_NODE_ID_INVALID) /* Static node */
+ name = node->name;
+ else /* Cloned node */
+ name = node->parent;
+
+ node_db = node_from_name(name);
+ if (node_db == NULL)
+ SET_ERR_JMP(ENOLINK, fail, "Node %s not found", name);
+ node->process = node_db->process;
+ }
+
+ return graph;
+fail:
+ return NULL;
+}
+
+static struct rte_graph *
+graph_mem_fixup_secondray(struct rte_graph *graph)
+{
+ if (graph == NULL || rte_eal_process_type() == RTE_PROC_PRIMARY)
+ return graph;
+
+ return graph_mem_fixup_node_ctx(graph);
+}
+
+struct rte_graph *
+rte_graph_lookup(const char *name)
+{
+ const struct rte_memzone *mz;
+ struct rte_graph *rc = NULL;
+
+ mz = rte_memzone_lookup(name);
+ if (mz)
+ rc = mz->addr;
+
+ return graph_mem_fixup_secondray(rc);
+}
+
rte_graph_t
rte_graph_create(const char *name, struct rte_graph_param *prm)
{
@@ -341,6 +389,76 @@ rte_graph_destroy(const char *graph_name)
return rc;
}
+rte_graph_t
+rte_graph_from_name(const char *name)
+{
+ struct graph *graph;
+
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (strncmp(graph->name, name, RTE_GRAPH_NAMESIZE) == 0)
+ return graph->id;
+
+ return RTE_GRAPH_ID_INVALID;
+}
+
+char *
+rte_graph_id_to_name(rte_graph_t id)
+{
+ struct graph *graph;
+
+ GRAPH_ID_CHECK(id);
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (graph->id == id)
+ return graph->name;
+
+fail:
+ return NULL;
+}
+
+struct rte_node *
+rte_graph_node_get(rte_graph_t gid, uint32_t nid)
+{
+ struct rte_node *node;
+ struct graph *graph;
+ rte_graph_off_t off;
+ rte_node_t count;
+
+ GRAPH_ID_CHECK(gid);
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (graph->id == gid) {
+ rte_graph_foreach_node(count, off, graph->graph,
+ node) {
+ if (node->id == nid)
+ return node;
+ }
+ break;
+ }
+fail:
+ return NULL;
+}
+
+struct rte_node *
+rte_graph_node_get_by_name(const char *graph_name, const char *node_name)
+{
+ struct rte_node *node;
+ struct graph *graph;
+ rte_graph_off_t off;
+ rte_node_t count;
+
+ STAILQ_FOREACH(graph, &graph_list, next)
+ if (!strncmp(graph->name, graph_name, RTE_GRAPH_NAMESIZE)) {
+ rte_graph_foreach_node(count, off, graph->graph,
+ node) {
+ if (!strncmp(node->name, node_name,
+ RTE_NODE_NAMESIZE))
+ return node;
+ }
+ break;
+ }
+
+ return NULL;
+}
+
void __rte_noinline
__rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
{
@@ -355,3 +473,16 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
node->size = size;
node->realloc_count++;
}
+
+rte_graph_t
+rte_graph_max_count(void)
+{
+ return graph_id;
+}
+
+RTE_INIT(rte_graph_init_log)
+{
+ rte_graph_logtype = rte_log_register("lib.graph");
+ if (rte_graph_logtype >= 0)
+ rte_log_set_level(rte_graph_logtype, RTE_LOG_INFO);
+}
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index dcbd78c02..5a2b13293 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -6,6 +6,14 @@ EXPERIMENTAL {
rte_graph_create;
rte_graph_destroy;
+ rte_graph_from_name;
+ rte_graph_id_to_name;
+ rte_graph_lookup;
+ rte_graph_list_dump;
+ rte_graph_max_count;
+ rte_graph_node_get;
+ rte_graph_node_get_by_name;
+
rte_node_clone;
rte_node_dump;
rte_node_edge_count;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 09/28] graph: implement Graphviz export
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (7 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 08/28] graph: implement graph operation APIs jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 10/28] graph: implement debug routines jerinj
` (20 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding API implementation support exporting the graph object to file.
This will export the graph to a file in Graphviz format.
It can be viewed in many viewers such as
https://dreampuf.github.io/GraphvizOnline/
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph.c | 54 ++++++++++++++++++++++++++
lib/librte_graph/rte_graph_version.map | 1 +
2 files changed, 55 insertions(+)
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index 7c6a7897d..e0c0b71a7 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -474,6 +474,60 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
node->realloc_count++;
}
+static int
+graph_to_dot(FILE *f, struct graph *graph)
+{
+ const char *src_edge_color = " [color=blue]\n";
+ const char *edge_color = "\n";
+ struct graph_node *graph_node;
+ char *node_name;
+ rte_edge_t i;
+ int rc;
+
+ rc = fprintf(f, "Digraph %s {\n\trankdir=LR;\n", graph->name);
+ if (rc < 0)
+ goto end;
+
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ node_name = graph_node->node->name;
+ for (i = 0; i < graph_node->node->nb_edges; i++) {
+ rc = fprintf(f, "\t\"%s\"->\"%s\"%s", node_name,
+ graph_node->adjacency_list[i]->node->name,
+ graph_node->node->flags & RTE_NODE_SOURCE_F
+ ? src_edge_color
+ : edge_color);
+ if (rc < 0)
+ goto end;
+ }
+ }
+ rc = fprintf(f, "}\n");
+ if (rc < 0)
+ goto end;
+
+ return 0;
+end:
+ rte_errno = EBADF;
+ return -rte_errno;
+}
+
+rte_graph_t
+rte_graph_export(const char *name, FILE *f)
+{
+ rte_graph_t rc = RTE_GRAPH_ID_INVALID;
+ struct graph *graph;
+
+ STAILQ_FOREACH(graph, &graph_list, next) {
+ if (strncmp(graph->name, name, RTE_GRAPH_NAMESIZE) == 0) {
+ rc = graph_to_dot(f, graph);
+ goto end;
+ }
+ }
+ rte_errno = ENOENT;
+end:
+ return rc;
+}
+
+
rte_graph_t
rte_graph_max_count(void)
{
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 5a2b13293..2797be044 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -6,6 +6,7 @@ EXPERIMENTAL {
rte_graph_create;
rte_graph_destroy;
+ rte_graph_export;
rte_graph_from_name;
rte_graph_id_to_name;
rte_graph_lookup;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 10/28] graph: implement debug routines
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (8 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 09/28] graph: implement Graphviz export jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 11/28] graph: implement stats support jerinj
` (19 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding implementation for graph specific API to dump the
Graph information to a file. This API will dump detailed internal
info about node objects and graph objects.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_graph/graph.c | 31 ++++++++++++++
lib/librte_graph/graph_debug.c | 59 ++++++++++++++++++++++++++
lib/librte_graph/graph_private.h | 13 ++++++
lib/librte_graph/rte_graph_version.map | 2 +
4 files changed, 105 insertions(+)
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index e0c0b71a7..e96363777 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -527,6 +527,37 @@ rte_graph_export(const char *name, FILE *f)
return rc;
}
+static void
+graph_scan_dump(FILE *f, rte_graph_t id, bool all)
+{
+ struct graph *graph;
+
+ RTE_VERIFY(f);
+ GRAPH_ID_CHECK(id);
+
+ STAILQ_FOREACH(graph, &graph_list, next) {
+ if (all == true) {
+ graph_dump(f, graph);
+ } else if (graph->id == id) {
+ graph_dump(f, graph);
+ return;
+ }
+ }
+fail:
+ return;
+}
+
+void
+rte_graph_dump(FILE *f, rte_graph_t id)
+{
+ graph_scan_dump(f, id, false);
+}
+
+void
+rte_graph_list_dump(FILE *f)
+{
+ graph_scan_dump(f, 0, true);
+}
rte_graph_t
rte_graph_max_count(void)
diff --git a/lib/librte_graph/graph_debug.c b/lib/librte_graph/graph_debug.c
index 75238e7ca..f8aea16ac 100644
--- a/lib/librte_graph/graph_debug.c
+++ b/lib/librte_graph/graph_debug.c
@@ -7,6 +7,26 @@
#include "graph_private.h"
+void
+graph_dump(FILE *f, struct graph *g)
+{
+ struct graph_node *graph_node;
+ rte_edge_t i = 0;
+
+ fprintf(f, "graph <%s>\n", g->name);
+ fprintf(f, " id=%" PRIu32 "\n", g->id);
+ fprintf(f, " cir_start=%" PRIu32 "\n", g->cir_start);
+ fprintf(f, " cir_mask=%" PRIu32 "\n", g->cir_mask);
+ fprintf(f, " addr=%p\n", g);
+ fprintf(f, " graph=%p\n", g->graph);
+ fprintf(f, " mem_sz=%zu\n", g->mem_sz);
+ fprintf(f, " node_count=%" PRIu32 "\n", g->node_count);
+ fprintf(f, " src_node_count=%" PRIu32 "\n", g->src_node_count);
+
+ STAILQ_FOREACH(graph_node, &g->node_list, next)
+ fprintf(f, " node[%d] <%s>\n", i++, graph_node->node->name);
+}
+
void
node_dump(FILE *f, struct node *n)
{
@@ -23,3 +43,42 @@ node_dump(FILE *f, struct node *n)
fprintf(f, " edge[%d] <%s>\n", i, n->next_nodes[i]);
}
+void
+rte_graph_obj_dump(FILE *f, struct rte_graph *g, bool all)
+{
+ rte_node_t count;
+ rte_graph_off_t off;
+ struct rte_node *n;
+ rte_edge_t i;
+
+ fprintf(f, "graph <%s> @ %p\n", g->name, g);
+ fprintf(f, " id=%" PRIu32 "\n", g->id);
+ fprintf(f, " head=%" PRId32 "\n", (int32_t)g->head);
+ fprintf(f, " tail=%" PRId32 "\n", (int32_t)g->tail);
+ fprintf(f, " cir_mask=0x%" PRIx32 "\n", g->cir_mask);
+ fprintf(f, " nb_nodes=%" PRId32 "\n", g->nb_nodes);
+ fprintf(f, " socket=%d\n", g->socket);
+ fprintf(f, " fence=0x%" PRIx64 "\n", g->fence);
+ fprintf(f, " nodes_start=0x%" PRIx32 "\n", g->nodes_start);
+ fprintf(f, " cir_start=%p\n", g->cir_start);
+
+ rte_graph_foreach_node(count, off, g, n) {
+ if (!all && n->idx == 0)
+ continue;
+ fprintf(f, " node[%d] <%s>\n", count, n->name);
+ fprintf(f, " fence=0x%" PRIx64 "\n", n->fence);
+ fprintf(f, " objs=%p\n", n->objs);
+ fprintf(f, " process=%p\n", n->process);
+ fprintf(f, " id=0x%" PRIx32 "\n", n->id);
+ fprintf(f, " offset=0x%" PRIx32 "\n", n->off);
+ fprintf(f, " nb_edges=%" PRId32 "\n", n->nb_edges);
+ fprintf(f, " realloc_count=%d\n", n->realloc_count);
+ fprintf(f, " size=%d\n", n->size);
+ fprintf(f, " idx=%d\n", n->idx);
+ fprintf(f, " total_objs=%" PRId64 "\n", n->total_objs);
+ fprintf(f, " total_calls=%" PRId64 "\n", n->total_calls);
+ for (i = 0; i < n->nb_edges; i++)
+ fprintf(f, " edge[%d] <%s>\n", i,
+ n->nodes[i]->name);
+ }
+}
diff --git a/lib/librte_graph/graph_private.h b/lib/librte_graph/graph_private.h
index 7fce52e00..f9a85c892 100644
--- a/lib/librte_graph/graph_private.h
+++ b/lib/librte_graph/graph_private.h
@@ -319,6 +319,19 @@ struct rte_node *graph_node_id_to_ptr(const struct rte_graph *graph,
struct rte_node *graph_node_name_to_ptr(const struct rte_graph *graph,
const char *node_name);
+/* Debug functions */
+/**
+ * @internal
+ *
+ * Dump internal graph object data.
+ *
+ * @param f
+ * FILE pointer to dump the data.
+ * @param g
+ * Pointer to the internal graph object.
+ */
+void graph_dump(FILE *f, struct graph *g);
+
/**
* @internal
*
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 2797be044..851f4772e 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -6,6 +6,7 @@ EXPERIMENTAL {
rte_graph_create;
rte_graph_destroy;
+ rte_graph_dump;
rte_graph_export;
rte_graph_from_name;
rte_graph_id_to_name;
@@ -14,6 +15,7 @@ EXPERIMENTAL {
rte_graph_max_count;
rte_graph_node_get;
rte_graph_node_get_by_name;
+ rte_graph_obj_dump;
rte_node_clone;
rte_node_dump;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 11/28] graph: implement stats support
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (9 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 10/28] graph: implement debug routines jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 12/28] graph: implement fastpath API routines jerinj
` (18 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding implementation for graph stats collection API. This API will
create a cluster for a specified node pattern and aggregate the node
runtime stats.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
lib/librte_graph/Makefile | 1 +
lib/librte_graph/graph_stats.c | 406 +++++++++++++++++++++++++
lib/librte_graph/meson.build | 2 +-
lib/librte_graph/rte_graph_version.map | 5 +
4 files changed, 413 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_graph/graph_stats.c
diff --git a/lib/librte_graph/Makefile b/lib/librte_graph/Makefile
index 7bfd7d51f..967c8d9bc 100644
--- a/lib/librte_graph/Makefile
+++ b/lib/librte_graph/Makefile
@@ -18,6 +18,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += node.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_ops.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_debug.c
+SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_stats.c
SRCS-$(CONFIG_RTE_LIBRTE_GRAPH) += graph_populate.c
# install header files
diff --git a/lib/librte_graph/graph_stats.c b/lib/librte_graph/graph_stats.c
new file mode 100644
index 000000000..125e08d73
--- /dev/null
+++ b/lib/librte_graph/graph_stats.c
@@ -0,0 +1,406 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <fnmatch.h>
+#include <stdbool.h>
+
+#include <rte_common.h>
+#include <rte_errno.h>
+#include <rte_malloc.h>
+
+#include "graph_private.h"
+
+/* Capture all graphs of cluster */
+struct cluster {
+ rte_graph_t nb_graphs;
+ rte_graph_t size;
+
+ struct graph **graphs;
+};
+
+/* Capture same node ID across cluster */
+struct cluster_node {
+ struct rte_graph_cluster_node_stats stat;
+ rte_node_t nb_nodes;
+
+ struct rte_node *nodes[];
+};
+
+struct rte_graph_cluster_stats {
+ /* Header */
+ rte_graph_cluster_stats_cb_t fn;
+ uint32_t cluster_node_size; /* Size of struct cluster_node */
+ rte_node_t max_nodes;
+ int socket_id;
+ void *cookie;
+ size_t sz;
+
+ struct cluster_node clusters[];
+} __rte_cache_aligned;
+
+#define boarder() \
+ fprintf(f, "+-------------------------------+---------------+--------" \
+ "-------+---------------+---------------+---------------+-" \
+ "----------+\n")
+
+static inline void
+print_banner(FILE *f)
+{
+ boarder();
+ fprintf(f, "%-32s%-16s%-16s%-16s%-16s%-16s%-16s\n", "|Node", "|calls",
+ "|objs", "|realloc_count", "|objs/call", "|objs/sec(10E6)",
+ "|cycles/call|");
+ boarder();
+}
+
+static inline void
+print_node(FILE *f, const struct rte_graph_cluster_node_stats *stat)
+{
+ double objs_per_call, objs_per_sec, cycles_per_call, ts_per_hz;
+ const uint64_t prev_calls = stat->prev_calls;
+ const uint64_t prev_objs = stat->prev_objs;
+ const uint64_t cycles = stat->cycles;
+ const uint64_t calls = stat->calls;
+ const uint64_t objs = stat->objs;
+ uint64_t call_delta;
+
+ call_delta = calls - prev_calls;
+ objs_per_call =
+ call_delta ? (double)((objs - prev_objs) / call_delta) : 0;
+ cycles_per_call =
+ call_delta ? (double)((cycles - stat->prev_cycles) / call_delta)
+ : 0;
+ ts_per_hz = (double)((stat->ts - stat->prev_ts) / stat->hz);
+ objs_per_sec = ts_per_hz ? (objs - prev_objs) / ts_per_hz : 0;
+ objs_per_sec /= 1000000;
+
+ fprintf(f,
+ "|%-31s|%-15" PRIu64 "|%-15" PRIu64 "|%-15" PRIu64
+ "|%-15.3f|%-15.6f|%-11.4f|\n",
+ stat->name, calls, objs, stat->realloc_count, objs_per_call,
+ objs_per_sec, cycles_per_call);
+}
+
+static int
+graph_cluster_stats_cb(bool is_first, bool is_last, void *cookie,
+ const struct rte_graph_cluster_node_stats *stat)
+{
+ FILE *f = cookie;
+
+ if (unlikely(is_first))
+ print_banner(f);
+ if (stat->objs)
+ print_node(f, stat);
+ if (unlikely(is_last))
+ boarder();
+
+ return 0;
+};
+
+static struct rte_graph_cluster_stats *
+stats_mem_init(struct cluster *cluster,
+ const struct rte_graph_cluster_stats_param *prm)
+{
+ size_t sz = sizeof(struct rte_graph_cluster_stats);
+ struct rte_graph_cluster_stats *stats;
+ rte_graph_cluster_stats_cb_t fn;
+ int socket_id = prm->socket_id;
+ uint32_t cluster_node_size;
+
+ /* Fix up callback */
+ fn = prm->fn;
+ if (fn == NULL)
+ fn = graph_cluster_stats_cb;
+
+ cluster_node_size = sizeof(struct cluster_node);
+ /* For a given cluster, max nodes will be the max number of graphs */
+ cluster_node_size += cluster->nb_graphs * sizeof(struct rte_node *);
+ cluster_node_size = RTE_ALIGN(cluster_node_size, RTE_CACHE_LINE_SIZE);
+
+ stats = realloc(NULL, sz);
+ memset(stats, 0, sz);
+ if (stats) {
+ stats->fn = fn;
+ stats->cluster_node_size = cluster_node_size;
+ stats->max_nodes = 0;
+ stats->socket_id = socket_id;
+ stats->cookie = prm->cookie;
+ stats->sz = sz;
+ }
+
+ return stats;
+}
+
+static int
+stats_mem_populate(struct rte_graph_cluster_stats **stats_in,
+ struct rte_graph *graph, struct graph_node *graph_node)
+{
+ struct rte_graph_cluster_stats *stats = *stats_in;
+ rte_node_t id = graph_node->node->id;
+ struct cluster_node *cluster;
+ struct rte_node *node;
+ rte_node_t count;
+
+ cluster = stats->clusters;
+
+ /* Iterate over cluster node array to find node ID match */
+ for (count = 0; count < stats->max_nodes; count++) {
+ /* Found an existing node in the reel */
+ if (cluster->stat.id == id) {
+ node = graph_node_id_to_ptr(graph, id);
+ if (node == NULL)
+ SET_ERR_JMP(
+ ENOENT, err,
+ "Failed to find node %s in graph %s",
+ graph_node->node->name, graph->name);
+
+ cluster->nodes[cluster->nb_nodes++] = node;
+ return 0;
+ }
+ cluster = RTE_PTR_ADD(cluster, stats->cluster_node_size);
+ }
+
+ /* Hey, it is a new node, allocate space for it in the reel */
+ stats = realloc(stats, stats->sz + stats->cluster_node_size);
+ if (stats == NULL)
+ SET_ERR_JMP(ENOMEM, err, "Realloc failed");
+
+ /* Clear the new struct cluster_node area */
+ cluster = RTE_PTR_ADD(stats, stats->sz),
+ memset(cluster, 0, stats->cluster_node_size);
+ memcpy(cluster->stat.name, graph_node->node->name, RTE_NODE_NAMESIZE);
+ cluster->stat.id = graph_node->node->id;
+ cluster->stat.hz = rte_get_timer_hz();
+ node = graph_node_id_to_ptr(graph, id);
+ if (node == NULL)
+ SET_ERR_JMP(ENOENT, err, "Failed to find node %s in graph %s",
+ graph_node->node->name, graph->name);
+ cluster->nodes[cluster->nb_nodes++] = node;
+
+ stats->sz += stats->cluster_node_size;
+ stats->max_nodes++;
+ *stats_in = stats;
+
+ return 0;
+err:
+ return -rte_errno;
+}
+
+static void
+stats_mem_fini(struct rte_graph_cluster_stats *stats)
+{
+ free(stats);
+}
+
+static void
+cluster_init(struct cluster *cluster)
+{
+ memset(cluster, 0, sizeof(*cluster));
+}
+
+static int
+cluster_add(struct cluster *cluster, struct graph *graph)
+{
+ rte_graph_t count;
+ size_t sz;
+
+ /* Skip the if graph is already added to cluster */
+ for (count = 0; count < cluster->nb_graphs; count++)
+ if (cluster->graphs[count] == graph)
+ return 0;
+
+ /* Expand the cluster if required to store graph objects */
+ if (cluster->nb_graphs + 1 > cluster->size) {
+ cluster->size = RTE_MAX(1, cluster->size * 2);
+ sz = sizeof(struct graph *) * cluster->size;
+ cluster->graphs = realloc(cluster->graphs, sz);
+ if (cluster->graphs == NULL)
+ SET_ERR_JMP(ENOMEM, free, "Failed to realloc");
+ }
+
+ /* Add graph to cluster */
+ cluster->graphs[cluster->nb_graphs++] = graph;
+ return 0;
+
+free:
+ return -rte_errno;
+}
+
+static void
+cluster_fini(struct cluster *cluster)
+{
+ if (cluster->graphs)
+ free(cluster->graphs);
+}
+
+static int
+expand_pattern_to_cluster(struct cluster *cluster, const char *pattern)
+{
+ struct graph_head *graph_head = graph_list_head_get();
+ struct graph *graph;
+ bool found = false;
+
+ /* Check for pattern match */
+ STAILQ_FOREACH(graph, graph_head, next) {
+ if (fnmatch(pattern, graph->name, 0) == 0) {
+ if (cluster_add(cluster, graph))
+ goto fail;
+ found = true;
+ }
+ }
+ if (found == false)
+ SET_ERR_JMP(EFAULT, fail, "Pattern %s graph not found",
+ pattern);
+
+ return 0;
+fail:
+ return -rte_errno;
+}
+
+struct rte_graph_cluster_stats *
+rte_graph_cluster_stats_create(const struct rte_graph_cluster_stats_param *prm)
+{
+ struct rte_graph_cluster_stats *stats, *rc = NULL;
+ struct graph_node *graph_node;
+ struct cluster cluster;
+ struct graph *graph;
+ const char *pattern;
+ rte_graph_t i;
+
+ /* Sanity checks */
+ if (!rte_graph_has_stats_feature())
+ SET_ERR_JMP(EINVAL, fail, "Stats feature is not enabled");
+
+ if (prm == NULL)
+ SET_ERR_JMP(EINVAL, fail, "Invalid param");
+
+ if (prm->graph_patterns == NULL || prm->nb_graph_patterns == 0)
+ SET_ERR_JMP(EINVAL, fail, "Invalid graph param");
+
+ cluster_init(&cluster);
+
+ graph_spinlock_lock();
+ /* Expand graph pattern and add the graph to the cluster */
+ for (i = 0; i < prm->nb_graph_patterns; i++) {
+ pattern = prm->graph_patterns[i];
+ if (expand_pattern_to_cluster(&cluster, pattern))
+ goto bad_pattern;
+ }
+
+ /* Alloc the stats memory */
+ stats = stats_mem_init(&cluster, prm);
+ if (stats == NULL)
+ SET_ERR_JMP(ENOMEM, bad_pattern, "Failed alloc stats memory");
+
+ /* Iterate over M(Graph) x N (Nodes in graph) */
+ for (i = 0; i < cluster.nb_graphs; i++) {
+ graph = cluster.graphs[i];
+ STAILQ_FOREACH(graph_node, &graph->node_list, next) {
+ struct rte_graph *graph_fp = graph->graph;
+ if (stats_mem_populate(&stats, graph_fp, graph_node))
+ goto realloc_fail;
+ }
+ }
+
+ /* Finally copy to hugepage memory to avoid pressure on rte_realloc */
+ rc = rte_malloc_socket(NULL, stats->sz, 0, stats->socket_id);
+ if (rc)
+ rte_memcpy(rc, stats, stats->sz);
+ else
+ SET_ERR_JMP(ENOMEM, realloc_fail, "rte_malloc failed");
+
+realloc_fail:
+ stats_mem_fini(stats);
+bad_pattern:
+ graph_spinlock_unlock();
+ cluster_fini(&cluster);
+fail:
+ return rc;
+}
+
+void
+rte_graph_cluster_stats_destroy(struct rte_graph_cluster_stats *stat)
+{
+ return rte_free(stat);
+}
+
+static inline void
+cluster_node_arregate_stats(struct cluster_node *cluster)
+{
+ uint64_t calls = 0, cycles = 0, objs = 0, realloc_count = 0;
+ struct rte_graph_cluster_node_stats *stat = &cluster->stat;
+ struct rte_node *node;
+ rte_node_t count;
+
+ for (count = 0; count < cluster->nb_nodes; count++) {
+ node = cluster->nodes[count];
+
+ calls += node->total_calls;
+ objs += node->total_objs;
+ cycles += node->total_cycles;
+ realloc_count += node->realloc_count;
+ }
+
+ stat->calls = calls;
+ stat->objs = objs;
+ stat->cycles = cycles;
+ stat->ts = rte_get_timer_cycles();
+ stat->realloc_count = realloc_count;
+}
+
+static inline void
+cluster_node_store_prev_stats(struct cluster_node *cluster)
+{
+ struct rte_graph_cluster_node_stats *stat = &cluster->stat;
+
+ stat->prev_ts = stat->ts;
+ stat->prev_calls = stat->calls;
+ stat->prev_objs = stat->objs;
+ stat->prev_cycles = stat->cycles;
+}
+
+void
+rte_graph_cluster_stats_get(struct rte_graph_cluster_stats *stat, bool skip_cb)
+{
+ struct cluster_node *cluster;
+ rte_node_t count;
+ int rc = 0;
+
+ cluster = stat->clusters;
+
+ for (count = 0; count < stat->max_nodes; count++) {
+ cluster_node_arregate_stats(cluster);
+ if (!skip_cb)
+ rc = stat->fn(!count, (count == stat->max_nodes - 1),
+ stat->cookie, &cluster->stat);
+ cluster_node_store_prev_stats(cluster);
+ if (rc)
+ break;
+ cluster = RTE_PTR_ADD(cluster, stat->cluster_node_size);
+ }
+}
+
+void
+rte_graph_cluster_stats_reset(struct rte_graph_cluster_stats *stat)
+{
+ struct cluster_node *cluster;
+ rte_node_t count;
+
+ cluster = stat->clusters;
+
+ for (count = 0; count < stat->max_nodes; count++) {
+ struct rte_graph_cluster_node_stats *node = &cluster->stat;
+
+ node->ts = 0;
+ node->calls = 0;
+ node->objs = 0;
+ node->cycles = 0;
+ node->prev_ts = 0;
+ node->prev_calls = 0;
+ node->prev_objs = 0;
+ node->prev_cycles = 0;
+ node->realloc_count = 0;
+ cluster = RTE_PTR_ADD(cluster, stat->cluster_node_size);
+ }
+}
diff --git a/lib/librte_graph/meson.build b/lib/librte_graph/meson.build
index fb203a5e2..929a17f84 100644
--- a/lib/librte_graph/meson.build
+++ b/lib/librte_graph/meson.build
@@ -4,7 +4,7 @@
name = 'graph'
-sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_populate.c')
+sources = files('node.c', 'graph.c', 'graph_ops.c', 'graph_debug.c', 'graph_stats.c', 'graph_populate.c')
headers = files('rte_graph.h', 'rte_graph_worker.h')
allow_experimental_apis = true
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index 851f4772e..adf55d406 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -17,6 +17,11 @@ EXPERIMENTAL {
rte_graph_node_get_by_name;
rte_graph_obj_dump;
+ rte_graph_cluster_stats_create;
+ rte_graph_cluster_stats_destroy;
+ rte_graph_cluster_stats_get;
+ rte_graph_cluster_stats_reset;
+
rte_node_clone;
rte_node_dump;
rte_node_edge_count;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 12/28] graph: implement fastpath API routines
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (10 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 11/28] graph: implement stats support jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 13/28] graph: add unit test case jerinj
` (17 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: John McNamara, Marko Kovacevic, Jerin Jacob, Kiran Kumar K
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, pbhagavatula,
ndabilpuram
From: Jerin Jacob <jerinj@marvell.com>
Adding implementation for rte_graph_walk() API. This will perform a walk
on the circular buffer and call the process function of each node
and collect the stats if stats collection is enabled.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
doc/api/doxy-api-index.md | 1 +
lib/librte_graph/graph.c | 16 +
lib/librte_graph/rte_graph_version.map | 10 +
lib/librte_graph/rte_graph_worker.h | 434 +++++++++++++++++++++++++
4 files changed, 461 insertions(+)
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 5cc50f750..fd2ff64d7 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -160,6 +160,7 @@ The public API headers are grouped by topics:
[port_in_action] (@ref rte_port_in_action.h)
[table_action] (@ref rte_table_action.h)
* [graph] (@ref rte_graph.h):
+ [graph_worker] (@ref rte_graph_worker.h)
- **basic**:
[approx fraction] (@ref rte_approx.h),
diff --git a/lib/librte_graph/graph.c b/lib/librte_graph/graph.c
index e96363777..d5d816c71 100644
--- a/lib/librte_graph/graph.c
+++ b/lib/librte_graph/graph.c
@@ -474,6 +474,22 @@ __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node)
node->realloc_count++;
}
+void __rte_noinline
+__rte_node_stream_alloc_size(struct rte_graph *graph, struct rte_node *node,
+ uint16_t req_size)
+{
+ uint16_t size = node->size;
+
+ RTE_VERIFY(size != UINT16_MAX);
+ /* Allocate double amount of size to avoid immediate realloc */
+ size = RTE_MIN(UINT16_MAX, RTE_MAX(RTE_GRAPH_BURST_SIZE, req_size * 2));
+ node->objs = rte_realloc_socket(node->objs, size * sizeof(void *),
+ RTE_CACHE_LINE_SIZE, graph->socket);
+ RTE_VERIFY(node->objs);
+ node->size = size;
+ node->realloc_count++;
+}
+
static int
graph_to_dot(FILE *f, struct graph *graph)
{
diff --git a/lib/librte_graph/rte_graph_version.map b/lib/librte_graph/rte_graph_version.map
index adf55d406..13b838752 100644
--- a/lib/librte_graph/rte_graph_version.map
+++ b/lib/librte_graph/rte_graph_version.map
@@ -3,6 +3,7 @@ EXPERIMENTAL {
__rte_node_register;
__rte_node_stream_alloc;
+ __rte_node_stream_alloc_size;
rte_graph_create;
rte_graph_destroy;
@@ -16,6 +17,7 @@ EXPERIMENTAL {
rte_graph_node_get;
rte_graph_node_get_by_name;
rte_graph_obj_dump;
+ rte_graph_walk;
rte_graph_cluster_stats_create;
rte_graph_cluster_stats_destroy;
@@ -28,10 +30,18 @@ EXPERIMENTAL {
rte_node_edge_get;
rte_node_edge_shrink;
rte_node_edge_update;
+ rte_node_enqueue;
+ rte_node_enqueue_x1;
+ rte_node_enqueue_x2;
+ rte_node_enqueue_x4;
+ rte_node_enqueue_next;
rte_node_from_name;
rte_node_id_to_name;
rte_node_list_dump;
rte_node_max_count;
+ rte_node_next_stream_get;
+ rte_node_next_stream_put;
+ rte_node_next_stream_move;
local: *;
};
diff --git a/lib/librte_graph/rte_graph_worker.h b/lib/librte_graph/rte_graph_worker.h
index a8133739d..a1bfc498b 100644
--- a/lib/librte_graph/rte_graph_worker.h
+++ b/lib/librte_graph/rte_graph_worker.h
@@ -101,6 +101,440 @@ struct rte_node {
__rte_experimental
void __rte_node_stream_alloc(struct rte_graph *graph, struct rte_node *node);
+/**
+ * @internal
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Allocate a stream with requested number of objects.
+ *
+ * If stream already exists then re-allocate it to a larger size.
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ * @param req_size
+ * Number of objects to be allocated.
+ */
+__rte_experimental
+void __rte_node_stream_alloc_size(struct rte_graph *graph,
+ struct rte_node *node, uint16_t req_size);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Perform graph walk on the circular buffer and invoke the process function
+ * of the nodes and collect the stats.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup function.
+ *
+ * @see rte_graph_lookup()
+ */
+__rte_experimental
+static inline void
+rte_graph_walk(struct rte_graph *graph)
+{
+ const rte_graph_off_t *cir_start = graph->cir_start;
+ const rte_node_t mask = graph->cir_mask;
+ uint32_t head = graph->head;
+ struct rte_node *node;
+ uint64_t start;
+ uint16_t rc;
+ void **objs;
+
+ /*
+ * Walk on the source node(s) ((cir_start - head) -> cir_start) and then
+ * on the pending streams (cir_start -> (cir_start + mask) -> cir_start)
+ * in a circular buffer fashion.
+ *
+ * +-----+ <= cir_start - head [number of source nodes]
+ * | |
+ * | ... | <= source nodes
+ * | |
+ * +-----+ <= cir_start [head = 0] [tail = 0]
+ * | |
+ * | ... | <= pending streams
+ * | |
+ * +-----+ <= cir_start + mask
+ */
+ while (likely(head != graph->tail)) {
+ node = RTE_PTR_ADD(graph, cir_start[(int32_t)head++]);
+ RTE_ASSERT(node->fence == RTE_GRAPH_FENCE);
+ objs = node->objs;
+ rte_prefetch0(objs);
+
+ if (rte_graph_has_stats_feature()) {
+ start = rte_rdtsc();
+ rc = node->process(graph, node, objs, node->idx);
+ node->total_cycles += rte_rdtsc() - start;
+ node->total_calls++;
+ node->total_objs += rc;
+ } else {
+ node->process(graph, node, objs, node->idx);
+ }
+ node->idx = 0;
+ head = likely((int32_t)head > 0) ? head & mask : head;
+ }
+ graph->tail = 0;
+}
+
+/* Fast path helper functions */
+
+/**
+ * @internal
+ *
+ * Enqueue a given node to the tail of the graph reel.
+ *
+ * @param graph
+ * Pointer Graph object.
+ * @param node
+ * Pointer to node object to be enqueued.
+ */
+static __rte_always_inline void
+__rte_node_enqueue_tail_update(struct rte_graph *graph, struct rte_node *node)
+{
+ uint32_t tail;
+
+ tail = graph->tail;
+ graph->cir_start[tail++] = node->off;
+ graph->tail = tail & graph->cir_mask;
+}
+
+/**
+ * @internal
+ *
+ * Enqueue sequence prologue function.
+ *
+ * Updates the node to tail of graph reel and resizes the number of objects
+ * available in the stream as needed.
+ *
+ * @param graph
+ * Pointer to the graph object.
+ * @param node
+ * Pointer to the node object.
+ * @param idx
+ * Index at which the object enqueue starts from.
+ * @param space
+ * Space required for the object enqueue.
+ */
+static __rte_always_inline void
+__rte_node_enqueue_prologue(struct rte_graph *graph, struct rte_node *node,
+ const uint16_t idx, const uint16_t space)
+{
+
+ /* Add to the pending stream list if the node is new */
+ if (idx == 0)
+ __rte_node_enqueue_tail_update(graph, node);
+
+ if (unlikely(node->size < (idx + space)))
+ __rte_node_stream_alloc(graph, node);
+}
+
+/**
+ * @internal
+ *
+ * Get the node pointer from current node edge id.
+ *
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Edge id of the required node.
+ *
+ * @return
+ * Pointer to the node denoted by the edge id.
+ */
+static __rte_always_inline struct rte_node *
+__rte_node_next_node_get(struct rte_node *node, rte_edge_t next)
+{
+ RTE_ASSERT(next < node->nb_edges);
+ RTE_ASSERT(node->fence == RTE_GRAPH_FENCE);
+ node = node->nodes[next];
+ RTE_ASSERT(node->fence == RTE_GRAPH_FENCE);
+
+ return node;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue the objs to next node for further processing and set
+ * the next node to pending state in the circular buffer.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to enqueue objs.
+ * @param objs
+ * Objs to enqueue.
+ * @param nb_objs
+ * Number of objs to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, void **objs, uint16_t nb_objs)
+{
+ node = __rte_node_next_node_get(node, next);
+ const uint16_t idx = node->idx;
+
+ __rte_node_enqueue_prologue(graph, node, idx, nb_objs);
+
+ rte_memcpy(&node->objs[idx], objs, nb_objs * sizeof(void *));
+ node->idx = idx + nb_objs;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue only one obj to next node for further processing and
+ * set the next node to pending state in the circular buffer.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to enqueue objs.
+ * @param obj
+ * Obj to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue_x1(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, void *obj)
+{
+ node = __rte_node_next_node_get(node, next);
+ uint16_t idx = node->idx;
+
+ __rte_node_enqueue_prologue(graph, node, idx, 1);
+
+ node->objs[idx++] = obj;
+ node->idx = idx;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue only two objs to next node for further processing and
+ * set the next node to pending state in the circular buffer.
+ * Same as rte_node_enqueue_x1 but enqueue two objs.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to enqueue objs.
+ * @param obj0
+ * Obj to enqueue.
+ * @param obj1
+ * Obj to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue_x2(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, void *obj0, void *obj1)
+{
+ node = __rte_node_next_node_get(node, next);
+ uint16_t idx = node->idx;
+
+ __rte_node_enqueue_prologue(graph, node, idx, 2);
+
+ node->objs[idx++] = obj0;
+ node->objs[idx++] = obj1;
+ node->idx = idx;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue only four objs to next node for further processing and
+ * set the next node to pending state in the circular buffer.
+ * Same as rte_node_enqueue_x1 but enqueue four objs.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to enqueue objs.
+ * @param obj0
+ * 1st obj to enqueue.
+ * @param obj1
+ * 2nd obj to enqueue.
+ * @param obj2
+ * 3rd obj to enqueue.
+ * @param obj3
+ * 4th obj to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue_x4(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, void *obj0, void *obj1, void *obj2,
+ void *obj3)
+{
+ node = __rte_node_next_node_get(node, next);
+ uint16_t idx = node->idx;
+
+ __rte_node_enqueue_prologue(graph, node, idx, 4);
+
+ node->objs[idx++] = obj0;
+ node->objs[idx++] = obj1;
+ node->objs[idx++] = obj2;
+ node->objs[idx++] = obj3;
+ node->idx = idx;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Enqueue objs to multiple next nodes for further processing and
+ * set the next nodes to pending state in the circular buffer.
+ * objs[i] will be enqueued to nexts[i].
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param nexts
+ * List of relative next node indices to enqueue objs.
+ * @param objs
+ * List of objs to enqueue.
+ * @param nb_objs
+ * Number of objs to enqueue.
+ */
+__rte_experimental
+static inline void
+rte_node_enqueue_next(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t *nexts, void **objs, uint16_t nb_objs)
+{
+ uint16_t i;
+
+ for (i = 0; i < nb_objs; i++)
+ rte_node_enqueue_x1(graph, node, nexts[i], objs[i]);
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get the stream of next node to enqueue the objs.
+ * Once done with the updating the objs, needs to call
+ * rte_node_next_stream_put to put the next node to pending state.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index to get stream.
+ * @param nb_objs
+ * Requested free size of the next stream.
+ *
+ * @return
+ * Valid next stream on success.
+ *
+ * @see rte_node_next_stream_put().
+ */
+__rte_experimental
+static inline void **
+rte_node_next_stream_get(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, uint16_t nb_objs)
+{
+ node = __rte_node_next_node_get(node, next);
+ const uint16_t idx = node->idx;
+ uint16_t free_space = node->size - idx;
+
+ if (unlikely(free_space < nb_objs))
+ __rte_node_stream_alloc_size(graph, node, nb_objs);
+
+ return &node->objs[idx];
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Put the next stream to pending state in the circular buffer
+ * for further processing. Should be invoked followed by
+ * rte_node_next_stream_get().
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param node
+ * Current node pointer.
+ * @param next
+ * Relative next node index..
+ * @param idx
+ * Number of objs updated in the stream after getting the stream using
+ * rte_node_next_stream_get.
+ *
+ * @see rte_node_next_stream_get().
+ */
+__rte_experimental
+static inline void
+rte_node_next_stream_put(struct rte_graph *graph, struct rte_node *node,
+ rte_edge_t next, uint16_t idx)
+{
+ if (unlikely(!idx))
+ return;
+
+ node = __rte_node_next_node_get(node, next);
+ if (node->idx == 0)
+ __rte_node_enqueue_tail_update(graph, node);
+
+ node->idx += idx;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Home run scenario, Enqueue all the objs of current node to next
+ * node in optimized way by swapping the streams of both nodes.
+ * Performs good when next node is already not in pending state.
+ * If next node is already in pending state then normal enqueue
+ * will be used.
+ *
+ * @param graph
+ * Graph pointer returned from rte_graph_lookup().
+ * @param src
+ * Current node pointer.
+ * @param next
+ * Relative next node index.
+ */
+__rte_experimental
+static inline void
+rte_node_next_stream_move(struct rte_graph *graph, struct rte_node *src,
+ rte_edge_t next)
+{
+ struct rte_node *dst = __rte_node_next_node_get(src, next);
+
+ /* Let swap the pointers if dst don't have valid objs */
+ if (likely(dst->idx == 0)) {
+ void **dobjs = dst->objs;
+ uint16_t dsz = dst->size;
+ dst->objs = src->objs;
+ dst->size = src->size;
+ src->objs = dobjs;
+ src->size = dsz;
+ dst->idx = src->idx;
+ __rte_node_enqueue_tail_update(graph, dst);
+ } else { /* Move the objects from src node to dst node */
+ rte_node_enqueue(graph, src, next, src->objs, src->idx);
+ }
+}
+
#ifdef __cplusplus
}
#endif
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 13/28] graph: add unit test case
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (11 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 12/28] graph: implement fastpath API routines jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 14/28] graph: add performance testcase jerinj
` (16 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark,
pbhagavatula, ndabilpuram
From: Kiran Kumar K <kirankumark@marvell.com>
Adding the unit test to test the functionality of node and graph APIs.
Testing includes registering a node, cloning a node, creating a graph,
perform graph walk, collecting stats and all node and graph debug APIs.
example command to test:
echo "graph_autotest" | sudo ./build/app/test/dpdk-test -c 0x30
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
app/test/Makefile | 6 +
app/test/meson.build | 11 +-
app/test/test_graph.c | 819 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 834 insertions(+), 2 deletions(-)
create mode 100644 app/test/test_graph.c
diff --git a/app/test/Makefile b/app/test/Makefile
index 1f080d162..ce2e08e12 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -221,6 +221,10 @@ SRCS-y += test_event_timer_adapter.c
SRCS-y += test_event_crypto_adapter.c
endif
+ifeq ($(CONFIG_RTE_LIBRTE_GRAPH), y)
+SRCS-y += test_graph.c
+endif
+
ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
SRCS-y += test_rawdev.c
endif
@@ -240,6 +244,8 @@ endif
CFLAGS += -DALLOW_EXPERIMENTAL_API
CFLAGS += -O3
+# Strict-aliasing rules are violated by uint8_t[] to context size casts.
+CFLAGS += -fno-strict-aliasing
CFLAGS += $(WERROR_FLAGS)
LDLIBS += -lm
diff --git a/app/test/meson.build b/app/test/meson.build
index 351d29cb6..0de16290e 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -51,6 +51,7 @@ test_sources = files('commands.c',
'test_fib6_perf.c',
'test_func_reentrancy.c',
'test_flow_classify.c',
+ 'test_graph.c',
'test_hash.c',
'test_hash_functions.c',
'test_hash_multiwriter.c',
@@ -151,7 +152,9 @@ test_deps = ['acl',
'rib',
'ring',
'stack',
- 'timer'
+ 'timer',
+ 'graph',
+ 'node'
]
# Each test is marked with flag true/false
@@ -362,6 +365,8 @@ endif
# specify -D_GNU_SOURCE unconditionally
cflags += '-D_GNU_SOURCE'
+# Strict-aliasing rules are violated by uint8_t[] to context size casts.
+cflags += '-fno-strict-aliasing'
test_dep_objs = []
if dpdk_conf.has('RTE_LIBRTE_COMPRESSDEV')
@@ -385,13 +390,15 @@ endforeach
test_dep_objs += cc.find_library('execinfo', required: false)
link_libs = []
+link_nodes = []
if get_option('default_library') == 'static'
link_libs = dpdk_drivers
+ link_nodes = dpdk_graph_nodes
endif
dpdk_test = executable('dpdk-test',
test_sources,
- link_whole: link_libs,
+ link_whole: link_libs + link_nodes,
dependencies: test_dep_objs,
c_args: [cflags, '-DALLOW_EXPERIMENTAL_API'],
install_rpath: driver_install_path,
diff --git a/app/test/test_graph.c b/app/test/test_graph.c
new file mode 100644
index 000000000..a90dc8f30
--- /dev/null
+++ b/app/test/test_graph.c
@@ -0,0 +1,819 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#include <assert.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <stdio.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_errno.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_mbuf.h>
+
+#include "test.h"
+
+static uint16_t test_node_worker_source(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+static uint16_t test_node0_worker(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+static uint16_t test_node1_worker(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+static uint16_t test_node2_worker(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+static uint16_t test_node3_worker(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs);
+
+#define MBUFF_SIZE 512
+#define MAX_NODES 4
+
+static struct rte_mbuf mbuf[MAX_NODES + 1][MBUFF_SIZE];
+static void *mbuf_p[MAX_NODES + 1][MBUFF_SIZE];
+static rte_graph_t graph_id;
+static uint64_t obj_stats[MAX_NODES + 1];
+static uint64_t fn_calls[MAX_NODES + 1];
+
+const char *node_patterns[] = {
+ "test_node_source1", "test_node00",
+ "test_node00-test_node11", "test_node00-test_node22",
+ "test_node00-test_node33",
+};
+
+const char *node_names[] = {
+ "test_node00",
+ "test_node00-test_node11",
+ "test_node00-test_node22",
+ "test_node00-test_node33",
+};
+
+struct test_node_register {
+ char name[RTE_NODE_NAMESIZE];
+ rte_node_process_t process;
+ uint16_t nb_edges;
+ const char *next_nodes[MAX_NODES];
+};
+
+typedef struct {
+ uint32_t idx;
+ struct test_node_register node;
+} test_node_t;
+
+typedef struct {
+ test_node_t test_node[MAX_NODES];
+} test_main_t;
+
+static test_main_t test_main = {
+ .test_node = {
+ {
+ .node = {
+ .name = "test_node00",
+ .process = test_node0_worker,
+ .nb_edges = 2,
+ .next_nodes = {"test_node00-"
+ "test_node11",
+ "test_node00-"
+ "test_node22"},
+ },
+ },
+ {
+ .node = {
+ .name = "test_node11",
+ .process = test_node1_worker,
+ .nb_edges = 1,
+ .next_nodes = {"test_node00-"
+ "test_node22"},
+ },
+ },
+ {
+ .node = {
+ .name = "test_node22",
+ .process = test_node2_worker,
+ .nb_edges = 1,
+ .next_nodes = {"test_node00-"
+ "test_node33"},
+ },
+ },
+ {
+ .node = {
+ .name = "test_node33",
+ .process = test_node3_worker,
+ .nb_edges = 1,
+ .next_nodes = {"test_node00"},
+ },
+ },
+ },
+};
+
+static int
+node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ RTE_SET_USED(graph);
+ *(uint32_t *)node->ctx = node->id;
+
+ return 0;
+}
+
+static struct rte_node_register test_node_source = {
+ .name = "test_node_source1",
+ .process = test_node_worker_source,
+ .flags = RTE_NODE_SOURCE_F,
+ .nb_edges = 2,
+ .init = node_init,
+ .next_nodes = {"test_node00", "test_node00-test_node11"},
+};
+RTE_NODE_REGISTER(test_node_source);
+
+static struct rte_node_register test_node0 = {
+ .name = "test_node00",
+ .process = test_node0_worker,
+ .init = node_init,
+};
+RTE_NODE_REGISTER(test_node0);
+
+uint16_t
+test_node_worker_source(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ uint32_t obj_node0 = rand() % 100, obj_node1;
+ test_main_t *tm = &test_main;
+ struct rte_mbuf *data;
+ void **next_stream;
+ rte_node_t next;
+ uint32_t i;
+
+ RTE_SET_USED(objs);
+ nb_objs = RTE_GRAPH_BURST_SIZE;
+
+ /* Prepare stream for next node 0 */
+ obj_node0 = nb_objs * obj_node0 * 0.01;
+ next = 0;
+ next_stream = rte_node_next_stream_get(graph, node, next, obj_node0);
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[0][i];
+ data->udata64 = ((uint64_t)tm->test_node[0].idx << 32) | i;
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ next_stream[i] = &mbuf[0][i];
+ }
+ rte_node_next_stream_put(graph, node, next, obj_node0);
+
+ /* Prepare stream for next node 1 */
+ obj_node1 = nb_objs - obj_node0;
+ next = 1;
+ next_stream = rte_node_next_stream_get(graph, node, next, obj_node1);
+ for (i = 0; i < obj_node1; i++) {
+ data = &mbuf[0][obj_node0 + i];
+ data->udata64 = ((uint64_t)tm->test_node[1].idx << 32) | i;
+ if ((i + 1) == obj_node1)
+ data->udata64 |= (1 << 16);
+ next_stream[i] = &mbuf[0][obj_node0 + i];
+ }
+
+ rte_node_next_stream_put(graph, node, next, obj_node1);
+ obj_stats[0] += nb_objs;
+ fn_calls[0] += 1;
+ return nb_objs;
+}
+
+uint16_t
+test_node0_worker(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ test_main_t *tm = &test_main;
+
+ if (*(uint32_t *)node->ctx == test_node0.id) {
+ uint32_t obj_node0 = rand() % 100, obj_node1;
+ struct rte_mbuf *data;
+ uint8_t second_pass = 0;
+ uint32_t count = 0;
+ uint32_t i;
+
+ obj_stats[1] += nb_objs;
+ fn_calls[1] += 1;
+
+ for (i = 0; i < nb_objs; i++) {
+ data = (struct rte_mbuf *)objs[i];
+ if ((data->udata64 >> 32) != tm->test_node[0].idx) {
+ printf("Data idx miss match at node 0, expected"
+ " = %u got = %u\n",
+ tm->test_node[0].idx,
+ (uint32_t)(data->udata64 >> 32));
+ goto end;
+ }
+
+ if ((data->udata64 & 0xffff) != (i - count)) {
+ printf("Expected buff count miss match at "
+ "node 0\n");
+ goto end;
+ }
+
+ if (data->udata64 & (0x1 << 16))
+ count = i + 1;
+ if (data->udata64 & (0x1 << 17))
+ second_pass = 1;
+ }
+
+ if (count != i) {
+ printf("Count mismatch at node 0\n");
+ goto end;
+ }
+
+ obj_node0 = nb_objs * obj_node0 * 0.01;
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[1][i];
+ data->udata64 =
+ ((uint64_t)tm->test_node[1].idx << 32) | i;
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ if (second_pass)
+ data->udata64 |= (1 << 17);
+ }
+ rte_node_enqueue(graph, node, 0, (void **)&mbuf_p[1][0],
+ obj_node0);
+
+ obj_node1 = nb_objs - obj_node0;
+ for (i = 0; i < obj_node1; i++) {
+ data = &mbuf[1][obj_node0 + i];
+ data->udata64 =
+ ((uint64_t)tm->test_node[2].idx << 32) | i;
+ if ((i + 1) == obj_node1)
+ data->udata64 |= (1 << 16);
+ if (second_pass)
+ data->udata64 |= (1 << 17);
+ }
+ rte_node_enqueue(graph, node, 1, (void **)&mbuf_p[1][obj_node0],
+ obj_node1);
+
+ } else if (*(uint32_t *)node->ctx == tm->test_node[1].idx) {
+ test_node1_worker(graph, node, objs, nb_objs);
+ } else if (*(uint32_t *)node->ctx == tm->test_node[2].idx) {
+ test_node2_worker(graph, node, objs, nb_objs);
+ } else if (*(uint32_t *)node->ctx == tm->test_node[3].idx) {
+ test_node3_worker(graph, node, objs, nb_objs);
+ } else {
+ printf("Unexpected node context\n");
+ }
+
+end:
+ return nb_objs;
+}
+
+uint16_t
+test_node1_worker(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ test_main_t *tm = &test_main;
+ uint8_t second_pass = 0;
+ uint32_t obj_node0 = 0;
+ struct rte_mbuf *data;
+ uint32_t count = 0;
+ uint32_t i;
+
+ obj_stats[2] += nb_objs;
+ fn_calls[2] += 1;
+ for (i = 0; i < nb_objs; i++) {
+ data = (struct rte_mbuf *)objs[i];
+ if ((data->udata64 >> 32) != tm->test_node[1].idx) {
+ printf("Data idx miss match at node 1, expected = %u"
+ " got = %u\n",
+ tm->test_node[1].idx,
+ (uint32_t)(data->udata64 >> 32));
+ goto end;
+ }
+
+ if ((data->udata64 & 0xffff) != (i - count)) {
+ printf("Expected buff count miss match at node 1\n");
+ goto end;
+ }
+
+ if (data->udata64 & (0x1 << 16))
+ count = i + 1;
+ if (data->udata64 & (0x1 << 17))
+ second_pass = 1;
+ }
+
+ if (count != i) {
+ printf("Count mismatch at node 1\n");
+ goto end;
+ }
+
+ obj_node0 = nb_objs;
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[2][i];
+ data->udata64 = ((uint64_t)tm->test_node[2].idx << 32) | i;
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ if (second_pass)
+ data->udata64 |= (1 << 17);
+ }
+ rte_node_enqueue(graph, node, 0, (void **)&mbuf_p[2][0], obj_node0);
+
+end:
+ return nb_objs;
+}
+
+uint16_t
+test_node2_worker(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ test_main_t *tm = &test_main;
+ uint8_t second_pass = 0;
+ struct rte_mbuf *data;
+ uint32_t count = 0;
+ uint32_t obj_node0;
+ uint32_t i;
+
+ obj_stats[3] += nb_objs;
+ fn_calls[3] += 1;
+ for (i = 0; i < nb_objs; i++) {
+ data = (struct rte_mbuf *)objs[i];
+ if ((data->udata64 >> 32) != tm->test_node[2].idx) {
+ printf("Data idx miss match at node 2, expected = %u"
+ " got = %u\n",
+ tm->test_node[2].idx,
+ (uint32_t)(data->udata64 >> 32));
+ goto end;
+ }
+
+ if ((data->udata64 & 0xffff) != (i - count)) {
+ printf("Expected buff count miss match at node 2\n");
+ goto end;
+ }
+
+ if (data->udata64 & (0x1 << 16))
+ count = i + 1;
+ if (data->udata64 & (0x1 << 17))
+ second_pass = 1;
+ }
+
+ if (count != i) {
+ printf("Count mismatch at node 2\n");
+ goto end;
+ }
+
+ if (!second_pass) {
+ obj_node0 = nb_objs;
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[3][i];
+ data->udata64 =
+ ((uint64_t)tm->test_node[3].idx << 32) | i;
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ }
+ rte_node_enqueue(graph, node, 0, (void **)&mbuf_p[3][0],
+ obj_node0);
+ }
+
+end:
+ return nb_objs;
+}
+
+uint16_t
+test_node3_worker(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ test_main_t *tm = &test_main;
+ uint8_t second_pass = 0;
+ struct rte_mbuf *data;
+ uint32_t count = 0;
+ uint32_t obj_node0;
+ uint32_t i;
+
+ obj_stats[4] += nb_objs;
+ fn_calls[4] += 1;
+ for (i = 0; i < nb_objs; i++) {
+ data = (struct rte_mbuf *)objs[i];
+ if ((data->udata64 >> 32) != tm->test_node[3].idx) {
+ printf("Data idx miss match at node 3, expected = %u"
+ " got = %u\n",
+ tm->test_node[3].idx,
+ (uint32_t)(data->udata64 >> 32));
+ goto end;
+ }
+
+ if ((data->udata64 & 0xffff) != (i - count)) {
+ printf("Expected buff count miss match at node 3\n");
+ goto end;
+ }
+
+ if (data->udata64 & (0x1 << 16))
+ count = i + 1;
+ if (data->udata64 & (0x1 << 17))
+ second_pass = 1;
+ }
+
+ if (count != i) {
+ printf("Count mismatch at node 3\n");
+ goto end;
+ }
+
+ if (second_pass) {
+ printf("Unexpected buffers are at node 3\n");
+ goto end;
+ } else {
+ obj_node0 = nb_objs * 2;
+ for (i = 0; i < obj_node0; i++) {
+ data = &mbuf[4][i];
+ data->udata64 =
+ ((uint64_t)tm->test_node[0].idx << 32) | i;
+ data->udata64 |= (1 << 17);
+ if ((i + 1) == obj_node0)
+ data->udata64 |= (1 << 16);
+ }
+ rte_node_enqueue(graph, node, 0, (void **)&mbuf_p[4][0],
+ obj_node0);
+ }
+
+end:
+ return nb_objs;
+}
+
+static int
+test_lookup_functions(void)
+{
+ test_main_t *tm = &test_main;
+ int i;
+
+ /* Verify the name with ID */
+ for (i = 1; i < MAX_NODES; i++) {
+ char *name = rte_node_id_to_name(tm->test_node[i].idx);
+ if (strcmp(name, node_names[i]) != 0) {
+ printf("Test node name verify by ID = %d failed "
+ "Expected = %s, got %s\n",
+ i, node_names[i], name);
+ return -1;
+ }
+ }
+
+ /* Verify by name */
+ for (i = 1; i < MAX_NODES; i++) {
+ uint32_t idx = rte_node_from_name(node_names[i]);
+ if (idx != tm->test_node[i].idx) {
+ printf("Test node ID verify by name = %s failed "
+ "Expected = %d, got %d\n",
+ node_names[i], tm->test_node[i].idx, idx);
+ return -1;
+ }
+ }
+
+ /* Verify edge count */
+ for (i = 1; i < MAX_NODES; i++) {
+ uint32_t count = rte_node_edge_count(tm->test_node[i].idx);
+ if (count != tm->test_node[i].node.nb_edges) {
+ printf("Test number of edges for node = %s failed Expected = %d, got = %d\n",
+ tm->test_node[i].node.name,
+ tm->test_node[i].node.nb_edges, count);
+ return -1;
+ }
+ }
+
+ /* Verify edge names */
+ for (i = 1; i < MAX_NODES; i++) {
+ uint32_t j, count;
+ char **next_edges;
+
+ count = rte_node_edge_get(tm->test_node[i].idx, NULL);
+ if (count != tm->test_node[i].node.nb_edges * sizeof(char *)) {
+ printf("Test number of edge count for node = %s failed Expected = %d, got = %d\n",
+ tm->test_node[i].node.name,
+ tm->test_node[i].node.nb_edges, count);
+ return -1;
+ }
+ next_edges = malloc(count);
+ count = rte_node_edge_get(tm->test_node[i].idx, next_edges);
+ if (count != tm->test_node[i].node.nb_edges) {
+ printf("Test number of edges for node = %s failed Expected = %d, got %d\n",
+ tm->test_node[i].node.name,
+ tm->test_node[i].node.nb_edges, count);
+ return -1;
+ }
+
+ for (j = 0; j < count; j++) {
+ if (strcmp(next_edges[j],
+ tm->test_node[i].node.next_nodes[j]) != 0) {
+ printf("Edge name miss match, expected = %s got = %s\n",
+ tm->test_node[i].node.next_nodes[j],
+ next_edges[j]);
+ return -1;
+ }
+ }
+ free(next_edges);
+ }
+
+ return 0;
+}
+
+static int
+test_node_clone(void)
+{
+ test_main_t *tm = &test_main;
+ uint32_t node_id, dummy_id;
+ int i;
+
+ node_id = rte_node_from_name("test_node00");
+ tm->test_node[0].idx = node_id;
+
+ /* Clone with same name, should fail */
+ dummy_id = rte_node_clone(node_id, "test_node00");
+ if (!rte_node_is_invalid(dummy_id)) {
+ printf("Got valid id when clone with same name, Expecting fail\n");
+ return -1;
+ }
+
+ for (i = 1; i < MAX_NODES; i++) {
+ tm->test_node[i].idx =
+ rte_node_clone(node_id, tm->test_node[i].node.name);
+ if (rte_node_is_invalid(tm->test_node[i].idx)) {
+ printf("Got invalid node id\n");
+ return -1;
+ }
+ }
+
+ /* Clone from cloned node should fail */
+ dummy_id = rte_node_clone(tm->test_node[1].idx, "dummy_node");
+ if (!rte_node_is_invalid(dummy_id)) {
+ printf("Got valid node id when cloning from cloned node, expected fail\n");
+ return -1;
+ }
+
+ return 0;
+}
+
+static int
+test_update_edges(void)
+{
+ test_main_t *tm = &test_main;
+ uint32_t node_id;
+ uint16_t count;
+ int i;
+
+ node_id = rte_node_from_name("test_node00");
+ count = rte_node_edge_update(node_id, 0,
+ tm->test_node[0].node.next_nodes,
+ tm->test_node[0].node.nb_edges);
+ if (count != tm->test_node[0].node.nb_edges) {
+ printf("Update edges failed expected: %d got = %d\n",
+ tm->test_node[0].node.nb_edges, count);
+ return -1;
+ }
+
+ for (i = 1; i < MAX_NODES; i++) {
+ count = rte_node_edge_update(tm->test_node[i].idx, 0,
+ tm->test_node[i].node.next_nodes,
+ tm->test_node[i].node.nb_edges);
+ if (count != tm->test_node[i].node.nb_edges) {
+ printf("Update edges failed expected: %d got = %d\n",
+ tm->test_node[i].node.nb_edges, count);
+ return -1;
+ }
+
+ count = rte_node_edge_shrink(tm->test_node[i].idx,
+ tm->test_node[i].node.nb_edges);
+ if (count != tm->test_node[i].node.nb_edges) {
+ printf("Shrink edges failed\n");
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+test_create_graph(void)
+{
+ static const char *node_patterns_dummy[] = {
+ "test_node_source1", "test_node00",
+ "test_node00-test_node11", "test_node00-test_node22",
+ "test_node00-test_node33", "test_node00-dummy_node",
+ };
+ struct rte_graph_param gconf = {
+ .socket_id = SOCKET_ID_ANY,
+ .nb_node_patterns = 6,
+ .node_patterns = node_patterns_dummy,
+ };
+ uint32_t dummy_node_id;
+ uint32_t node_id;
+
+ node_id = rte_node_from_name("test_node00");
+ dummy_node_id = rte_node_clone(node_id, "dummy_node");
+ if (rte_node_is_invalid(dummy_node_id)) {
+ printf("Got invalid node id\n");
+ return -1;
+ }
+
+ graph_id = rte_graph_create("worker0", &gconf);
+ if (graph_id != RTE_GRAPH_ID_INVALID) {
+ printf("Graph creation success with isolated node, expected graph creation fail\n");
+ return -1;
+ }
+
+ gconf.nb_node_patterns = 5;
+ gconf.node_patterns = node_patterns;
+ graph_id = rte_graph_create("worker0", &gconf);
+ if (graph_id == RTE_GRAPH_ID_INVALID) {
+ printf("Graph creation failed with error = %d\n", rte_errno);
+ return -1;
+ }
+ return 0;
+}
+
+static int
+test_graph_walk(void)
+{
+ struct rte_graph *graph = rte_graph_lookup("worker0");
+ int i;
+
+ if (!graph) {
+ printf("Graph lookup failed\n");
+ return -1;
+ }
+
+ for (i = 0; i < 5; i++)
+ rte_graph_walk(graph);
+ return 0;
+}
+
+static int
+test_graph_lookup_functions(void)
+{
+ test_main_t *tm = &test_main;
+ struct rte_node *node;
+ int i;
+
+ for (i = 0; i < MAX_NODES; i++) {
+ node = rte_graph_node_get(graph_id, tm->test_node[i].idx);
+ if (!node) {
+ printf("rte_graph_node_get, failed for node = %d\n",
+ tm->test_node[i].idx);
+ return -1;
+ }
+
+ if (tm->test_node[i].idx != node->id) {
+ printf("Node id didn't match, expected = %d got = %d\n",
+ tm->test_node[i].idx, node->id);
+ return 0;
+ }
+
+ if (strncmp(node->name, node_names[i], RTE_NODE_NAMESIZE)) {
+ printf("Node name didn't match, expected = %s got %s\n",
+ node_names[i], node->name);
+ return -1;
+ }
+ }
+
+ for (i = 0; i < MAX_NODES; i++) {
+ node = rte_graph_node_get_by_name("worker0", node_names[i]);
+ if (!node) {
+ printf("rte_graph_node_get, failed for node = %d\n",
+ tm->test_node[i].idx);
+ return -1;
+ }
+
+ if (tm->test_node[i].idx != node->id) {
+ printf("Node id didn't match, expected = %d got = %d\n",
+ tm->test_node[i].idx, node->id);
+ return 0;
+ }
+
+ if (strncmp(node->name, node_names[i], RTE_NODE_NAMESIZE)) {
+ printf("Node name didn't match, expected = %s got %s\n",
+ node_names[i], node->name);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+graph_cluster_stats_cb_t(bool is_first, bool is_last, void *cookie,
+ const struct rte_graph_cluster_node_stats *st)
+{
+ int i;
+
+ RTE_SET_USED(is_first);
+ RTE_SET_USED(is_last);
+ RTE_SET_USED(cookie);
+
+ for (i = 0; i < MAX_NODES + 1; i++) {
+ rte_node_t id = rte_node_from_name(node_patterns[i]);
+ if (id == st->id) {
+ if (obj_stats[i] != st->objs) {
+ printf("Obj count miss match for node = %s expected = %"PRId64", got=%"PRId64"\n",
+ node_patterns[i], obj_stats[i],
+ st->objs);
+ return -1;
+ }
+
+ if (fn_calls[i] != st->calls) {
+ printf("Func call miss match for node = %s expected = %"PRId64", got = %"PRId64"\n",
+ node_patterns[i], fn_calls[i],
+ st->calls);
+ return -1;
+ }
+ }
+ }
+ return 0;
+}
+
+static int
+test_print_stats(void)
+{
+ struct rte_graph_cluster_stats_param s_param;
+ struct rte_graph_cluster_stats *stats;
+ const char *pattern = "worker0";
+
+ if (!rte_graph_has_stats_feature())
+ return 0;
+
+ /* Prepare stats object */
+ memset(&s_param, 0, sizeof(s_param));
+ s_param.f = stdout;
+ s_param.socket_id = SOCKET_ID_ANY;
+ s_param.graph_patterns = &pattern;
+ s_param.nb_graph_patterns = 1;
+ s_param.fn = graph_cluster_stats_cb_t;
+
+ stats = rte_graph_cluster_stats_create(&s_param);
+ if (stats == NULL) {
+ printf("Unable to get stats\n");
+ return -1;
+ }
+ /* Clear screen and move to top left */
+ rte_graph_cluster_stats_get(stats, 0);
+ rte_graph_cluster_stats_destroy(stats);
+
+ return 0;
+}
+
+static int
+graph_setup(void)
+{
+ int i, j;
+
+ for (i = 0; i <= MAX_NODES; i++) {
+ for (j = 0; j < MBUFF_SIZE; j++)
+ mbuf_p[i][j] = &mbuf[i][j];
+ }
+ if (test_node_clone()) {
+ printf("test_node_clone: fail\n");
+ return -1;
+ }
+ printf("test_node_clone: pass\n");
+
+ return 0;
+}
+
+static void
+graph_teardown(void)
+{
+ rte_graph_t id;
+
+ id = rte_graph_destroy("worker0");
+ if (id == RTE_GRAPH_ID_INVALID)
+ printf("Graph Destroy failed\n");
+}
+
+static struct unit_test_suite graph_testsuite = {
+ .suite_name = "Graph library test suite",
+ .setup = graph_setup,
+ .teardown = graph_teardown,
+ .unit_test_cases = {
+ TEST_CASE(test_update_edges),
+ TEST_CASE(test_lookup_functions),
+ TEST_CASE(test_create_graph),
+ TEST_CASE(test_graph_lookup_functions),
+ TEST_CASE(test_graph_walk),
+ TEST_CASE(test_print_stats),
+ TEST_CASES_END(), /**< NULL terminate unit test array */
+ },
+};
+
+static int
+graph_autotest_fn(void)
+{
+ return unit_test_suite_runner(&graph_testsuite);
+}
+
+REGISTER_TEST_COMMAND(graph_autotest, graph_autotest_fn);
+
+static int
+test_node_list_dump(void)
+{
+ rte_node_list_dump(stdout);
+
+ return TEST_SUCCESS;
+}
+REGISTER_TEST_COMMAND(node_list_dump, test_node_list_dump);
+
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 14/28] graph: add performance testcase
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (12 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 13/28] graph: add unit test case jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 15/28] node: add log infra and null node jerinj
` (15 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark,
pbhagavatula, ndabilpuram
From: Pavan Nikhilesh <pbhagavatula@marvell.com>
Add unit test framework to create and test performance of various graph
models.
example command to test:
echo "graph_perf_autotest" | sudo ./build/app/test/dpdk-test -c 0x30
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
app/test/Makefile | 1 +
app/test/meson.build | 1 +
app/test/test_graph_perf.c | 1057 ++++++++++++++++++++++++++++++++++++
3 files changed, 1059 insertions(+)
create mode 100644 app/test/test_graph_perf.c
diff --git a/app/test/Makefile b/app/test/Makefile
index ce2e08e12..77276f300 100644
--- a/app/test/Makefile
+++ b/app/test/Makefile
@@ -223,6 +223,7 @@ endif
ifeq ($(CONFIG_RTE_LIBRTE_GRAPH), y)
SRCS-y += test_graph.c
+SRCS-y += test_graph_perf.c
endif
ifeq ($(CONFIG_RTE_LIBRTE_RAWDEV),y)
diff --git a/app/test/meson.build b/app/test/meson.build
index 0de16290e..728d20c1f 100644
--- a/app/test/meson.build
+++ b/app/test/meson.build
@@ -52,6 +52,7 @@ test_sources = files('commands.c',
'test_func_reentrancy.c',
'test_flow_classify.c',
'test_graph.c',
+ 'test_graph_perf.c',
'test_hash.c',
'test_hash_functions.c',
'test_hash_multiwriter.c',
diff --git a/app/test/test_graph_perf.c b/app/test/test_graph_perf.c
new file mode 100644
index 000000000..a629f1e35
--- /dev/null
+++ b/app/test/test_graph_perf.c
@@ -0,0 +1,1057 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#include <inttypes.h>
+#include <signal.h>
+#include <stdio.h>
+#include <unistd.h>
+
+#include <rte_common.h>
+#include <rte_cycles.h>
+#include <rte_errno.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_lcore.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+
+#include "test.h"
+
+#define TEST_GRAPH_PERF_MZ "graph_perf_data"
+#define TEST_GRAPH_SRC_NAME "test_graph_perf_source"
+#define TEST_GRAPH_SRC_BRST_ONE_NAME "test_graph_perf_source_one"
+#define TEST_GRAPH_WRK_NAME "test_graph_perf_worker"
+#define TEST_GRAPH_SNK_NAME "test_graph_perf_sink"
+
+#define SOURCES(map) RTE_DIM(map)
+#define STAGES(map) RTE_DIM(map)
+#define NODES_PER_STAGE(map) RTE_DIM(map[0])
+#define SINKS(map) RTE_DIM(map[0])
+
+#define MAX_EDGES_PER_NODE 7
+
+struct test_node_data {
+ uint8_t node_id;
+ uint8_t is_sink;
+ uint8_t next_nodes[MAX_EDGES_PER_NODE];
+ uint8_t next_percentage[MAX_EDGES_PER_NODE];
+};
+
+struct test_graph_perf {
+ uint16_t nb_nodes;
+ rte_graph_t graph_id;
+ struct test_node_data *node_data;
+};
+
+struct graph_lcore_data {
+ uint8_t done;
+ rte_graph_t graph_id;
+};
+
+static struct test_node_data *
+graph_get_node_data(struct test_graph_perf *graph_data, rte_node_t id)
+{
+ struct test_node_data *node_data = NULL;
+ int i;
+
+ for (i = 0; i < graph_data->nb_nodes; i++)
+ if (graph_data->node_data[i].node_id == id) {
+ node_data = &graph_data->node_data[i];
+ break;
+ }
+
+ return node_data;
+}
+
+static int
+test_node_ctx_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ struct test_graph_perf *graph_data;
+ struct test_node_data *node_data;
+ const struct rte_memzone *mz;
+ rte_node_t nid = node->id;
+ rte_edge_t edge = 0;
+ int i;
+
+ RTE_SET_USED(graph);
+
+ mz = rte_memzone_lookup(TEST_GRAPH_PERF_MZ);
+ graph_data = mz->addr;
+ node_data = graph_get_node_data(graph_data, nid);
+ node->ctx[0] = node->nb_edges;
+ for (i = 0; i < node->nb_edges && !node_data->is_sink; i++, edge++) {
+ node->ctx[i + 1] = edge;
+ node->ctx[i + 9] = node_data->next_percentage[i];
+ }
+
+ return 0;
+}
+
+/* Source node function */
+static uint16_t
+test_perf_node_worker_source(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ uint16_t count;
+ int i;
+
+ RTE_SET_USED(objs);
+ RTE_SET_USED(nb_objs);
+
+ /* Create a proportional stream for every next */
+ for (i = 0; i < node->ctx[0]; i++) {
+ count = (node->ctx[i + 9] * RTE_GRAPH_BURST_SIZE) / 100;
+ rte_node_next_stream_get(graph, node, node->ctx[i + 1], count);
+ rte_node_next_stream_put(graph, node, node->ctx[i + 1], count);
+ }
+
+ return RTE_GRAPH_BURST_SIZE;
+}
+
+static struct rte_node_register test_graph_perf_source = {
+ .name = TEST_GRAPH_SRC_NAME,
+ .process = test_perf_node_worker_source,
+ .flags = RTE_NODE_SOURCE_F,
+ .init = test_node_ctx_init,
+};
+
+RTE_NODE_REGISTER(test_graph_perf_source);
+
+static uint16_t
+test_perf_node_worker_source_burst_one(struct rte_graph *graph,
+ struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ uint16_t count;
+ int i;
+
+ RTE_SET_USED(objs);
+ RTE_SET_USED(nb_objs);
+
+ /* Create a proportional stream for every next */
+ for (i = 0; i < node->ctx[0]; i++) {
+ count = (node->ctx[i + 9]) / 100;
+ rte_node_next_stream_get(graph, node, node->ctx[i + 1], count);
+ rte_node_next_stream_put(graph, node, node->ctx[i + 1], count);
+ }
+
+ return 1;
+}
+
+static struct rte_node_register test_graph_perf_source_burst_one = {
+ .name = TEST_GRAPH_SRC_BRST_ONE_NAME,
+ .process = test_perf_node_worker_source_burst_one,
+ .flags = RTE_NODE_SOURCE_F,
+ .init = test_node_ctx_init,
+};
+
+RTE_NODE_REGISTER(test_graph_perf_source_burst_one);
+
+/* Worker node function */
+static uint16_t
+test_perf_node_worker(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ uint16_t next = 0;
+ uint16_t enq = 0;
+ uint16_t count;
+ int i;
+
+ /* Move stream for single next node */
+ if (node->ctx[0] == 1) {
+ rte_node_next_stream_move(graph, node, node->ctx[1]);
+ return nb_objs;
+ }
+
+ /* Enqueue objects to next nodes proportionally */
+ for (i = 0; i < node->ctx[0]; i++) {
+ next = node->ctx[i + 1];
+ count = (node->ctx[i + 9] * nb_objs) / 100;
+ enq += count;
+ while (count) {
+ switch (count & (4 - 1)) {
+ case 0:
+ rte_node_enqueue_x4(graph, node, next, objs[0],
+ objs[1], objs[2], objs[3]);
+ objs += 4;
+ count -= 4;
+ break;
+ case 1:
+ rte_node_enqueue_x1(graph, node, next, objs[0]);
+ objs += 1;
+ count -= 1;
+ break;
+ case 2:
+ rte_node_enqueue_x2(graph, node, next, objs[0],
+ objs[1]);
+ objs += 2;
+ count -= 2;
+ break;
+ case 3:
+ rte_node_enqueue_x2(graph, node, next, objs[0],
+ objs[1]);
+ rte_node_enqueue_x1(graph, node, next, objs[0]);
+ objs += 3;
+ count -= 3;
+ break;
+ }
+ }
+ }
+
+ if (enq != nb_objs)
+ rte_node_enqueue(graph, node, next, objs, nb_objs - enq);
+
+ return nb_objs;
+}
+
+static struct rte_node_register test_graph_perf_worker = {
+ .name = TEST_GRAPH_WRK_NAME,
+ .process = test_perf_node_worker,
+ .init = test_node_ctx_init,
+};
+
+RTE_NODE_REGISTER(test_graph_perf_worker);
+
+/* Last node in graph a.k.a sink node */
+static uint16_t
+test_perf_node_sink(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ RTE_SET_USED(graph);
+ RTE_SET_USED(node);
+ RTE_SET_USED(objs);
+ RTE_SET_USED(nb_objs);
+
+ return nb_objs;
+}
+
+static struct rte_node_register test_graph_perf_sink = {
+ .name = TEST_GRAPH_SNK_NAME,
+ .process = test_perf_node_sink,
+ .init = test_node_ctx_init,
+};
+
+RTE_NODE_REGISTER(test_graph_perf_sink);
+
+static int
+graph_perf_setup(void)
+{
+ if (rte_lcore_count() < 2) {
+ printf("Test requires at least 2 lcores\n");
+ return TEST_SKIPPED;
+ }
+
+ return 0;
+}
+
+static void
+graph_perf_teardown(void)
+{
+}
+
+static inline rte_node_t
+graph_node_get(const char *pname, char *nname)
+{
+ rte_node_t pnode_id = rte_node_from_name(pname);
+ char lookup_name[RTE_NODE_NAMESIZE];
+ rte_node_t node_id;
+
+ snprintf(lookup_name, RTE_NODE_NAMESIZE, "%s-%s", pname, nname);
+ node_id = rte_node_from_name(lookup_name);
+
+ if (node_id != RTE_NODE_ID_INVALID) {
+ if (rte_node_edge_count(node_id))
+ rte_node_edge_shrink(node_id, 0);
+ return node_id;
+ }
+
+ return rte_node_clone(pnode_id, nname);
+}
+
+static uint16_t
+graph_node_count_edges(uint32_t stage, uint16_t node, uint16_t nodes_per_stage,
+ uint8_t edge_map[][nodes_per_stage][nodes_per_stage],
+ char *ename[], struct test_node_data *node_data,
+ rte_node_t **node_map)
+{
+ uint8_t total_percent = 0;
+ uint16_t edges = 0;
+ int i;
+
+ for (i = 0; i < nodes_per_stage && edges < MAX_EDGES_PER_NODE; i++) {
+ if (edge_map[stage + 1][i][node]) {
+ ename[edges] = malloc(sizeof(char) * RTE_NODE_NAMESIZE);
+ snprintf(ename[edges], RTE_NODE_NAMESIZE, "%s",
+ rte_node_id_to_name(node_map[stage + 1][i]));
+ node_data->next_nodes[edges] = node_map[stage + 1][i];
+ node_data->next_percentage[edges] =
+ edge_map[stage + 1][i][node];
+ edges++;
+ total_percent += edge_map[stage + 1][i][node];
+ }
+ }
+
+ if (edges >= MAX_EDGES_PER_NODE || (edges && total_percent != 100)) {
+ for (i = 0; i < edges; i++)
+ free(ename[i]);
+ return RTE_EDGE_ID_INVALID;
+ }
+
+ return edges;
+}
+
+static int
+graph_init(const char *gname, uint8_t nb_srcs, uint8_t nb_sinks,
+ uint32_t stages, uint16_t nodes_per_stage,
+ uint8_t src_map[][nodes_per_stage], uint8_t snk_map[][nb_sinks],
+ uint8_t edge_map[][nodes_per_stage][nodes_per_stage],
+ uint8_t burst_one)
+{
+ struct test_graph_perf *graph_data;
+ char nname[RTE_NODE_NAMESIZE / 2];
+ struct test_node_data *node_data;
+ char *ename[nodes_per_stage];
+ struct rte_graph_param gconf;
+ const struct rte_memzone *mz;
+ uint8_t total_percent = 0;
+ rte_node_t *src_nodes;
+ rte_node_t *snk_nodes;
+ rte_node_t **node_map;
+ char **node_patterns;
+ rte_graph_t graph_id;
+ rte_edge_t edges;
+ rte_edge_t count;
+ uint32_t i, j, k;
+
+ mz = rte_memzone_reserve(TEST_GRAPH_PERF_MZ,
+ sizeof(struct test_graph_perf), 0, 0);
+ if (mz == NULL) {
+ printf("Failed to allocate graph common memory\n");
+ return -ENOMEM;
+ }
+
+ graph_data = mz->addr;
+ graph_data->nb_nodes = 0;
+ graph_data->node_data =
+ malloc(sizeof(struct test_node_data) *
+ (nb_srcs + nb_sinks + stages * nodes_per_stage));
+ if (graph_data->node_data == NULL) {
+ printf("Failed to reserve memzone for graph data\n");
+ goto memzone_free;
+ }
+
+ node_patterns = malloc(sizeof(char *) *
+ (nb_srcs + nb_sinks + stages * nodes_per_stage));
+ if (node_patterns == NULL) {
+ printf("Failed to reserve memory for node patterns\n");
+ goto data_free;
+ }
+
+ src_nodes = malloc(sizeof(rte_node_t) * nb_srcs);
+ if (src_nodes == NULL) {
+ printf("Failed to reserve memory for src nodes\n");
+ goto pattern_free;
+ }
+
+ snk_nodes = malloc(sizeof(rte_node_t) * nb_sinks);
+ if (snk_nodes == NULL) {
+ printf("Failed to reserve memory for snk nodes\n");
+ goto src_free;
+ }
+
+ node_map = malloc(sizeof(rte_node_t *) * stages +
+ sizeof(rte_node_t) * nodes_per_stage * stages);
+ if (node_map == NULL) {
+ printf("Failed to reserve memory for node map\n");
+ goto snk_free;
+ }
+
+ /* Setup the Graph */
+ for (i = 0; i < stages; i++) {
+ node_map[i] =
+ (rte_node_t *)(node_map + stages) + nodes_per_stage * i;
+ for (j = 0; j < nodes_per_stage; j++) {
+ total_percent = 0;
+ for (k = 0; k < nodes_per_stage; k++)
+ total_percent += edge_map[i][j][k];
+ if (!total_percent)
+ continue;
+ node_patterns[graph_data->nb_nodes] =
+ malloc(RTE_NODE_NAMESIZE);
+ if (node_patterns[graph_data->nb_nodes] == NULL) {
+ printf("Failed to create memory for pattern\n");
+ goto pattern_name_free;
+ }
+
+ /* Clone a worker node */
+ snprintf(nname, sizeof(nname), "%d-%d", i, j);
+ node_map[i][j] =
+ graph_node_get(TEST_GRAPH_WRK_NAME, nname);
+ if (node_map[i][j] == RTE_NODE_ID_INVALID) {
+ printf("Failed to create node[%s]\n", nname);
+ graph_data->nb_nodes++;
+ goto pattern_name_free;
+ }
+ snprintf(node_patterns[graph_data->nb_nodes],
+ RTE_NODE_NAMESIZE, "%s",
+ rte_node_id_to_name(node_map[i][j]));
+ node_data =
+ &graph_data->node_data[graph_data->nb_nodes];
+ node_data->node_id = node_map[i][j];
+ node_data->is_sink = false;
+ graph_data->nb_nodes++;
+ }
+ }
+
+ for (i = 0; i < stages - 1; i++) {
+ for (j = 0; j < nodes_per_stage; j++) {
+ /* Count edges i.e connections of worker node to next */
+ node_data =
+ graph_get_node_data(graph_data, node_map[i][j]);
+ edges = graph_node_count_edges(i, j, nodes_per_stage,
+ edge_map, ename,
+ node_data, node_map);
+ if (edges == RTE_EDGE_ID_INVALID) {
+ printf("Invalid edge configuration\n");
+ goto pattern_name_free;
+ }
+ if (!edges)
+ continue;
+
+ /* Connect a node in stage 'i' to nodes
+ * in stage 'i + 1' with edges.
+ */
+ count = rte_node_edge_update(
+ node_map[i][j], 0,
+ (const char **)(uintptr_t)ename, edges);
+ for (k = 0; k < edges; k++)
+ free(ename[k]);
+ if (count != edges) {
+ printf("Couldn't add edges %d %d\n", edges,
+ count);
+ goto pattern_name_free;
+ }
+ }
+ }
+
+ /* Setup Source nodes */
+ for (i = 0; i < nb_srcs; i++) {
+ edges = 0;
+ total_percent = 0;
+ node_patterns[graph_data->nb_nodes] = malloc(RTE_NODE_NAMESIZE);
+ if (node_patterns[graph_data->nb_nodes] == NULL) {
+ printf("Failed to create memory for pattern\n");
+ goto pattern_name_free;
+ }
+ /* Clone a source node */
+ snprintf(nname, sizeof(nname), "%d", i);
+ src_nodes[i] =
+ graph_node_get(burst_one ? TEST_GRAPH_SRC_BRST_ONE_NAME
+ : TEST_GRAPH_SRC_NAME,
+ nname);
+ if (src_nodes[i] == RTE_NODE_ID_INVALID) {
+ printf("Failed to create node[%s]\n", nname);
+ graph_data->nb_nodes++;
+ goto pattern_name_free;
+ }
+ snprintf(node_patterns[graph_data->nb_nodes], RTE_NODE_NAMESIZE,
+ "%s", rte_node_id_to_name(src_nodes[i]));
+ node_data = &graph_data->node_data[graph_data->nb_nodes];
+ node_data->node_id = src_nodes[i];
+ node_data->is_sink = false;
+ graph_data->nb_nodes++;
+
+ /* Prepare next node list to connect to */
+ for (j = 0; j < nodes_per_stage; j++) {
+ if (!src_map[i][j])
+ continue;
+ ename[edges] = malloc(sizeof(char) * RTE_NODE_NAMESIZE);
+ snprintf(ename[edges], RTE_NODE_NAMESIZE, "%s",
+ rte_node_id_to_name(node_map[0][j]));
+ node_data->next_nodes[edges] = node_map[0][j];
+ node_data->next_percentage[edges] = src_map[i][j];
+ edges++;
+ total_percent += src_map[i][j];
+ }
+
+ if (!edges)
+ continue;
+ if (edges >= MAX_EDGES_PER_NODE || total_percent != 100) {
+ printf("Invalid edge configuration\n");
+ for (j = 0; j < edges; j++)
+ free(ename[j]);
+ goto pattern_name_free;
+ }
+
+ /* Connect to list of next nodes using edges */
+ count = rte_node_edge_update(src_nodes[i], 0,
+ (const char **)(uintptr_t)ename,
+ edges);
+ for (k = 0; k < edges; k++)
+ free(ename[k]);
+ if (count != edges) {
+ printf("Couldn't add edges %d %d\n", edges, count);
+ goto pattern_name_free;
+ }
+ }
+
+ /* Setup Sink nodes */
+ for (i = 0; i < nb_sinks; i++) {
+ node_patterns[graph_data->nb_nodes] = malloc(RTE_NODE_NAMESIZE);
+ if (node_patterns[graph_data->nb_nodes] == NULL) {
+ printf("Failed to create memory for pattern\n");
+ goto pattern_name_free;
+ }
+
+ /* Clone a sink node */
+ snprintf(nname, sizeof(nname), "%d", i);
+ snk_nodes[i] = graph_node_get(TEST_GRAPH_SNK_NAME, nname);
+ if (snk_nodes[i] == RTE_NODE_ID_INVALID) {
+ printf("Failed to create node[%s]\n", nname);
+ graph_data->nb_nodes++;
+ goto pattern_name_free;
+ }
+ snprintf(node_patterns[graph_data->nb_nodes], RTE_NODE_NAMESIZE,
+ "%s", rte_node_id_to_name(snk_nodes[i]));
+ node_data = &graph_data->node_data[graph_data->nb_nodes];
+ node_data->node_id = snk_nodes[i];
+ node_data->is_sink = true;
+ graph_data->nb_nodes++;
+ }
+
+ /* Connect last stage worker nodes to sink nodes */
+ for (i = 0; i < nodes_per_stage; i++) {
+ edges = 0;
+ total_percent = 0;
+ node_data = graph_get_node_data(graph_data,
+ node_map[stages - 1][i]);
+ /* Prepare list of sink nodes to connect to */
+ for (j = 0; j < nb_sinks; j++) {
+ if (!snk_map[i][j])
+ continue;
+ ename[edges] = malloc(sizeof(char) * RTE_NODE_NAMESIZE);
+ snprintf(ename[edges], RTE_NODE_NAMESIZE, "%s",
+ rte_node_id_to_name(snk_nodes[j]));
+ node_data->next_nodes[edges] = snk_nodes[j];
+ node_data->next_percentage[edges] = snk_map[i][j];
+ edges++;
+ total_percent += snk_map[i][j];
+ }
+ if (!edges)
+ continue;
+ if (edges >= MAX_EDGES_PER_NODE || total_percent != 100) {
+ printf("Invalid edge configuration\n");
+ for (j = 0; j < edges; j++)
+ free(ename[i]);
+ goto pattern_name_free;
+ }
+
+ /* Connect a worker node to a list of sink nodes */
+ count = rte_node_edge_update(node_map[stages - 1][i], 0,
+ (const char **)(uintptr_t)ename,
+ edges);
+ for (k = 0; k < edges; k++)
+ free(ename[k]);
+ if (count != edges) {
+ printf("Couldn't add edges %d %d\n", edges, count);
+ goto pattern_name_free;
+ }
+ }
+
+ /* Create a Graph */
+ gconf.socket_id = SOCKET_ID_ANY;
+ gconf.nb_node_patterns = graph_data->nb_nodes;
+ gconf.node_patterns = (const char **)(uintptr_t)node_patterns;
+
+ graph_id = rte_graph_create(gname, &gconf);
+ if (graph_id == RTE_GRAPH_ID_INVALID) {
+ printf("Graph creation failed with error = %d\n", rte_errno);
+ goto pattern_name_free;
+ }
+ graph_data->graph_id = graph_id;
+
+ for (i = 0; i < graph_data->nb_nodes; i++)
+ free(node_patterns[i]);
+ free(snk_nodes);
+ free(src_nodes);
+ free(node_patterns);
+ return 0;
+
+pattern_name_free:
+ for (i = 0; i < graph_data->nb_nodes; i++)
+ free(node_patterns[i]);
+snk_free:
+ free(snk_nodes);
+src_free:
+ free(src_nodes);
+pattern_free:
+ free(node_patterns);
+data_free:
+ free(graph_data->node_data);
+memzone_free:
+ rte_memzone_free(mz);
+ return -ENOMEM;
+}
+
+/* Worker thread function */
+static int
+_graph_perf_wrapper(void *args)
+{
+ struct graph_lcore_data *data = args;
+ struct rte_graph *graph;
+
+ /* Lookup graph */
+ graph = rte_graph_lookup(rte_graph_id_to_name(data->graph_id));
+
+ /* Graph walk until done */
+ while (!data->done)
+ rte_graph_walk(graph);
+
+ return 0;
+}
+
+static int
+measure_perf_get(rte_graph_t graph_id)
+{
+ const char *pattern = rte_graph_id_to_name(graph_id);
+ uint32_t lcore_id = rte_get_next_lcore(-1, 1, 0);
+ struct rte_graph_cluster_stats_param param;
+ struct rte_graph_cluster_stats *stats;
+ struct graph_lcore_data *data;
+
+ data = rte_zmalloc("Graph_perf", sizeof(struct graph_lcore_data),
+ RTE_CACHE_LINE_SIZE);
+ data->graph_id = graph_id;
+ data->done = 0;
+
+ /* Run graph worker thread function */
+ rte_eal_remote_launch(_graph_perf_wrapper, data, lcore_id);
+
+ /* Collect stats for few msecs */
+ if (rte_graph_has_stats_feature()) {
+ memset(¶m, 0, sizeof(param));
+ param.f = stdout;
+ param.socket_id = SOCKET_ID_ANY;
+ param.graph_patterns = &pattern;
+ param.nb_graph_patterns = 1;
+
+ stats = rte_graph_cluster_stats_create(¶m);
+ if (stats == NULL) {
+ printf("Failed to create stats\n");
+ return -ENOMEM;
+ }
+
+ rte_delay_ms(3E2);
+ rte_graph_cluster_stats_get(stats, true);
+ rte_delay_ms(1E3);
+ rte_graph_cluster_stats_get(stats, false);
+ rte_graph_cluster_stats_destroy(stats);
+ } else
+ rte_delay_ms(1E3);
+
+ data->done = 1;
+ rte_eal_wait_lcore(lcore_id);
+
+ return 0;
+}
+
+static inline void
+graph_fini(void)
+{
+ const struct rte_memzone *mz = rte_memzone_lookup(TEST_GRAPH_PERF_MZ);
+ struct test_graph_perf *graph_data;
+
+ if (mz == NULL)
+ return;
+ graph_data = mz->addr;
+
+ rte_graph_destroy(rte_graph_id_to_name(graph_data->graph_id));
+ free(graph_data->node_data);
+ rte_memzone_free(rte_memzone_lookup(TEST_GRAPH_PERF_MZ));
+}
+
+static int
+measure_perf(void)
+{
+ const struct rte_memzone *mz;
+ struct test_graph_perf *graph_data;
+
+ mz = rte_memzone_lookup(TEST_GRAPH_PERF_MZ);
+ graph_data = mz->addr;
+
+ return measure_perf_get(graph_data->graph_id);
+}
+
+static inline int
+graph_hr_4s_1n_1src_1snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_hr_4s_1n_1src_1snk_brst_one(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_hr_4s_1n_2src_1snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_hr_4s_1n_1src_2snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_tree_4s_4n_1src_4snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_reverse_tree_3s_4n_1src_1snk(void)
+{
+ return measure_perf();
+}
+
+static inline int
+graph_parallel_tree_5s_4n_4src_4snk(void)
+{
+ return measure_perf();
+}
+
+/* Graph Topology
+ * nodes per stage: 1
+ * stages: 4
+ * src: 1
+ * sink: 1
+ */
+static inline int
+graph_init_hr(void)
+{
+ uint8_t edge_map[][1][1] = {
+ { {100} },
+ { {100} },
+ { {100} },
+ { {100} },
+ };
+ uint8_t src_map[][1] = { {100} };
+ uint8_t snk_map[][1] = { {100} };
+
+ return graph_init("graph_hr", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 1
+ * stages: 4
+ * src: 1
+ * sink: 1
+ */
+static inline int
+graph_init_hr_brst_one(void)
+{
+ uint8_t edge_map[][1][1] = {
+ { {100} },
+ { {100} },
+ { {100} },
+ { {100} },
+ };
+ uint8_t src_map[][1] = { {100} };
+ uint8_t snk_map[][1] = { {100} };
+
+ return graph_init("graph_hr", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 1);
+}
+
+/* Graph Topology
+ * nodes per stage: 1
+ * stages: 4
+ * src: 2
+ * sink: 1
+ */
+static inline int
+graph_init_hr_multi_src(void)
+{
+ uint8_t edge_map[][1][1] = {
+ { {100} },
+ { {100} },
+ { {100} },
+ { {100} },
+ };
+ uint8_t src_map[][1] = {
+ {100}, {100}
+ };
+ uint8_t snk_map[][1] = { {100} };
+
+ return graph_init("graph_hr", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 1
+ * stages: 4
+ * src: 1
+ * sink: 2
+ */
+static inline int
+graph_init_hr_multi_snk(void)
+{
+ uint8_t edge_map[][1][1] = {
+ { {100} },
+ { {100} },
+ { {100} },
+ { {100} },
+ };
+ uint8_t src_map[][1] = { {100} };
+ uint8_t snk_map[][2] = { {50, 50} };
+
+ return graph_init("graph_hr", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 4
+ * stages: 4
+ * src: 1
+ * sink: 4
+ */
+static inline int
+graph_init_tree(void)
+{
+ uint8_t edge_map[][4][4] = {
+ {
+ {100, 0, 0, 0},
+ {0, 0, 0, 0},
+ {0, 0, 0, 0},
+ {0, 0, 0, 0}
+ },
+ {
+ {50, 0, 0, 0},
+ {50, 0, 0, 0},
+ {0, 0, 0, 0},
+ {0, 0, 0, 0}
+ },
+ {
+ {33, 33, 0, 0},
+ {34, 34, 0, 0},
+ {33, 33, 0, 0},
+ {0, 0, 0, 0}
+ },
+ {
+ {25, 25, 25, 0},
+ {25, 25, 25, 0},
+ {25, 25, 25, 0},
+ {25, 25, 25, 0}
+ }
+ };
+ uint8_t src_map[][4] = { {100, 0, 0, 0} };
+ uint8_t snk_map[][4] = {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ };
+
+ return graph_init("graph_full_split", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 4
+ * stages: 3
+ * src: 1
+ * sink: 1
+ */
+static inline int
+graph_init_reverse_tree(void)
+{
+ uint8_t edge_map[][4][4] = {
+ {
+ {25, 25, 25, 25},
+ {25, 25, 25, 25},
+ {25, 25, 25, 25},
+ {25, 25, 25, 25}
+ },
+ {
+ {33, 33, 33, 33},
+ {33, 33, 33, 33},
+ {34, 34, 34, 34},
+ {0, 0, 0, 0}
+ },
+ {
+ {50, 50, 50, 0},
+ {50, 50, 50, 0},
+ {0, 0, 0, 0},
+ {0, 0, 0, 0}
+ },
+ };
+ uint8_t src_map[][4] = { {25, 25, 25, 25} };
+ uint8_t snk_map[][1] = { {100}, {100}, {0}, {0} };
+
+ return graph_init("graph_full_split", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/* Graph Topology
+ * nodes per stage: 4
+ * stages: 5
+ * src: 4
+ * sink: 4
+ */
+static inline int
+graph_init_parallel_tree(void)
+{
+ uint8_t edge_map[][4][4] = {
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ },
+ };
+ uint8_t src_map[][4] = {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ };
+ uint8_t snk_map[][4] = {
+ {100, 0, 0, 0},
+ {0, 100, 0, 0},
+ {0, 0, 100, 0},
+ {0, 0, 0, 100}
+ };
+
+ return graph_init("graph_parallel", SOURCES(src_map), SINKS(snk_map),
+ STAGES(edge_map), NODES_PER_STAGE(edge_map), src_map,
+ snk_map, edge_map, 0);
+}
+
+/** Graph Creation cheat sheet
+ * edge_map -> dictates graph flow from worker stage 0 to worker stage n-1.
+ * src_map -> dictates source nodes enqueue percentage to worker stage 0.
+ * snk_map -> dictates stage n-1 enqueue percentage to sink.
+ *
+ * Layout:
+ * edge_map[<nb_stages>][<nodes_per_stg>][<nodes_in_nxt_stg = nodes_per_stg>]
+ * src_map[<nb_sources>][<nodes_in_stage0 = nodes_per_stage>]
+ * snk_map[<nodes_in_stage(n-1) = nodes_per_stage>][<nb_sinks>]
+ *
+ * The last array dictates the percentage of received objs to enqueue to next
+ * stage.
+ *
+ * Note: edge_map[][0][] will always be unused as it will receive from source
+ *
+ * Example:
+ * Graph:
+ * http://bit.ly/2PqbqOy
+ * Each stage(n) connects to all nodes in the next stage in decreasing
+ * order.
+ * Since we can't resize the edge_map dynamically we get away by creating
+ * dummy nodes and assigning 0 percentages.
+ * Max nodes across all stages = 4
+ * stages = 3
+ * nb_src = 1
+ * nb_snk = 1
+ * // Stages
+ * edge_map[][4][4] = {
+ * // Nodes per stage
+ * {
+ * {25, 25, 25, 25},
+ * {25, 25, 25, 25},
+ * {25, 25, 25, 25},
+ * {25, 25, 25, 25}
+ * }, // This will be unused.
+ * {
+ * // Nodes enabled in current stage + prev stage enq %
+ * {33, 33, 33, 33},
+ * {33, 33, 33, 33},
+ * {34, 34, 34, 34},
+ * {0, 0, 0, 0}
+ * },
+ * {
+ * {50, 50, 50, 0},
+ * {50, 50, 50, 0},
+ * {0, 0, 0, 0},
+ * {0, 0, 0, 0}
+ * },
+ * };
+ * Above, each stage tells how much it should receive from previous except
+ * from stage_0.
+ *
+ * src_map[][4] = { {25, 25, 25, 25} };
+ * Here, we tell each source the % it has to send to stage_0 nodes. In
+ * case we want 2 source node we can declare as
+ * src_map[][4] = { {25, 25, 25, 25}, {25, 25, 25, 25} };
+ *
+ * snk_map[][1] = { {100}, {100}, {0}, {0} }
+ * Here, we tell stage - 1 nodes how much to enqueue to sink_0.
+ * If we have 2 sinks we can do as follows
+ * snk_map[][2] = { {50, 50}, {50, 50}, {0, 0}, {0, 0} }
+ */
+
+static struct unit_test_suite graph_perf_testsuite = {
+ .suite_name = "Graph library performance test suite",
+ .setup = graph_perf_setup,
+ .teardown = graph_perf_teardown,
+ .unit_test_cases = {
+ TEST_CASE_ST(graph_init_hr, graph_fini,
+ graph_hr_4s_1n_1src_1snk),
+ TEST_CASE_ST(graph_init_hr_brst_one, graph_fini,
+ graph_hr_4s_1n_1src_1snk_brst_one),
+ TEST_CASE_ST(graph_init_hr_multi_src, graph_fini,
+ graph_hr_4s_1n_2src_1snk),
+ TEST_CASE_ST(graph_init_hr_multi_snk, graph_fini,
+ graph_hr_4s_1n_1src_2snk),
+ TEST_CASE_ST(graph_init_tree, graph_fini,
+ graph_tree_4s_4n_1src_4snk),
+ TEST_CASE_ST(graph_init_reverse_tree, graph_fini,
+ graph_reverse_tree_3s_4n_1src_1snk),
+ TEST_CASE_ST(graph_init_parallel_tree, graph_fini,
+ graph_parallel_tree_5s_4n_4src_4snk),
+ TEST_CASES_END(), /**< NULL terminate unit test array */
+ },
+};
+
+static int
+test_graph_perf_func(void)
+{
+ return unit_test_suite_runner(&graph_perf_testsuite);
+}
+
+REGISTER_TEST_COMMAND(graph_perf_autotest, test_graph_perf_func);
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 15/28] node: add log infra and null node
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (13 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 14/28] graph: add performance testcase jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 16/28] node: add ethdev Rx node jerinj
` (14 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Thomas Monjalon, John McNamara, Marko Kovacevic,
Nithin Dabilpuram, Pavan Nikhilesh, Bruce Richardson
Cc: dev, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add log infra for node specific logging.
Also, add null rte_node that just ignores all the objects
directed to it.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
MAINTAINERS | 5 +++++
config/common_base | 5 +++++
doc/api/doxy-api.conf.in | 1 +
lib/Makefile | 3 +++
lib/librte_node/Makefile | 22 ++++++++++++++++++++++
lib/librte_node/log.c | 14 ++++++++++++++
lib/librte_node/meson.build | 8 ++++++++
lib/librte_node/node_private.h | 22 ++++++++++++++++++++++
lib/librte_node/null.c | 23 +++++++++++++++++++++++
lib/librte_node/rte_node_version.map | 6 ++++++
lib/meson.build | 5 ++++-
meson.build | 1 +
mk/rte.app.mk | 1 +
13 files changed, 115 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_node/Makefile
create mode 100644 lib/librte_node/log.c
create mode 100644 lib/librte_node/meson.build
create mode 100644 lib/librte_node/node_private.h
create mode 100644 lib/librte_node/null.c
create mode 100644 lib/librte_node/rte_node_version.map
diff --git a/MAINTAINERS b/MAINTAINERS
index bc7085983..c1acaedab 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1474,6 +1474,11 @@ M: Jerin Jacob <jerinj@marvell.com>
M: Kiran Kumar K <kirankumark@marvell.com>
F: lib/librte_graph/
+Nodes - EXPERIMENTAL
+M: Nithin Dabilpuram <ndabilpuram@marvell.com>
+M: Pavan Nikhilesh <pbhagavatula@marvell.com>
+F: lib/librte_node/
+
Test Applications
-----------------
diff --git a/config/common_base b/config/common_base
index 32f982136..1ed5187dc 100644
--- a/config/common_base
+++ b/config/common_base
@@ -1081,6 +1081,11 @@ CONFIG_RTE_LIBRTE_GRAPH=y
CONFIG_RTE_GRAPH_BURST_SIZE=256
CONFIG_RTE_LIBRTE_GRAPH_STATS=y
+#
+# Compile librte_node
+#
+CONFIG_RTE_LIBRTE_NODE=y
+
#
# Compile the test application
#
diff --git a/doc/api/doxy-api.conf.in b/doc/api/doxy-api.conf.in
index 759a7213e..1d4f1a37d 100644
--- a/doc/api/doxy-api.conf.in
+++ b/doc/api/doxy-api.conf.in
@@ -49,6 +49,7 @@ INPUT = @TOPDIR@/doc/api/doxy-api-index.md \
@TOPDIR@/lib/librte_mempool \
@TOPDIR@/lib/librte_meter \
@TOPDIR@/lib/librte_metrics \
+ @TOPDIR@/lib/librte_node \
@TOPDIR@/lib/librte_net \
@TOPDIR@/lib/librte_pci \
@TOPDIR@/lib/librte_pdump \
diff --git a/lib/Makefile b/lib/Makefile
index 1f572b659..50d61a338 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -122,6 +122,9 @@ DEPDIRS-librte_rcu := librte_eal
DIRS-$(CONFIG_RTE_LIBRTE_GRAPH) += librte_graph
DEPDIRS-librte_graph := librte_eal
+DIRS-$(CONFIG_RTE_LIBRTE_NODE) += librte_node
+DEPDIRS-librte_node := librte_graph librte_lpm librte_ethdev librte_mbuf
+
ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
DIRS-$(CONFIG_RTE_LIBRTE_KNI) += librte_kni
endif
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
new file mode 100644
index 000000000..dbc8e1d44
--- /dev/null
+++ b/lib/librte_node/Makefile
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+#
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+# library name
+LIB = librte_node.a
+
+CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
+CFLAGS += $(WERROR_FLAGS)
+# Strict-aliasing rules are violated by uint8_t[] to context size casts.
+CFLAGS += -fno-strict-aliasing
+LDLIBS += -lrte_eal -lrte_graph
+
+EXPORT_MAP := rte_node_version.map
+
+# all source are stored in SRCS-y
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
+
+include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/log.c b/lib/librte_node/log.c
new file mode 100644
index 000000000..f035f91e8
--- /dev/null
+++ b/lib/librte_node/log.c
@@ -0,0 +1,14 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include "node_private.h"
+
+int rte_node_logtype;
+
+RTE_INIT(rte_node_init_log)
+{
+ rte_node_logtype = rte_log_register("lib.node");
+ if (rte_node_logtype >= 0)
+ rte_log_set_level(rte_node_logtype, RTE_LOG_INFO);
+}
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
new file mode 100644
index 000000000..a97813ad4
--- /dev/null
+++ b/lib/librte_node/meson.build
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+sources = files('null.c', 'log.c')
+allow_experimental_apis = true
+# Strict-aliasing rules are violated by uint8_t[] to context size casts.
+cflags += '-fno-strict-aliasing'
+deps += ['graph']
diff --git a/lib/librte_node/node_private.h b/lib/librte_node/node_private.h
new file mode 100644
index 000000000..f30902a94
--- /dev/null
+++ b/lib/librte_node/node_private.h
@@ -0,0 +1,22 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __NODE_PRIVATE_H__
+#define __NODE_PRIVATE_H__
+
+#include <rte_common.h>
+#include <rte_log.h>
+
+extern int rte_node_logtype;
+#define NODE_LOG(level, node_name, ...) \
+ rte_log(RTE_LOG_##level, rte_node_logtype, \
+ RTE_FMT("NODE %s: %s():%u " RTE_FMT_HEAD(__VA_ARGS__, ) "\n", \
+ node_name, __func__, __LINE__, \
+ RTE_FMT_TAIL(__VA_ARGS__, )))
+
+#define node_err(node_name, ...) NODE_LOG(ERR, node_name, __VA_ARGS__)
+#define node_info(node_name, ...) NODE_LOG(INFO, node_name, __VA_ARGS__)
+#define node_dbg(node_name, ...) NODE_LOG(DEBUG, node_name, __VA_ARGS__)
+
+#endif /* __NODE_PRIVATE_H__ */
diff --git a/lib/librte_node/null.c b/lib/librte_node/null.c
new file mode 100644
index 000000000..c7cd8b6df
--- /dev/null
+++ b/lib/librte_node/null.c
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_graph.h>
+
+static uint16_t
+null(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ RTE_SET_USED(node);
+ RTE_SET_USED(objs);
+ RTE_SET_USED(graph);
+
+ return nb_objs;
+}
+
+static struct rte_node_register null_node = {
+ .name = "null",
+ .process = null,
+};
+
+RTE_NODE_REGISTER(null_node);
diff --git a/lib/librte_node/rte_node_version.map b/lib/librte_node/rte_node_version.map
new file mode 100644
index 000000000..f87163bb9
--- /dev/null
+++ b/lib/librte_node/rte_node_version.map
@@ -0,0 +1,6 @@
+EXPERIMENTAL {
+ global:
+
+ rte_node_logtype;
+ local: *;
+};
diff --git a/lib/meson.build b/lib/meson.build
index c43d86bb9..147129b0b 100644
--- a/lib/meson.build
+++ b/lib/meson.build
@@ -30,7 +30,7 @@ libraries = [
# add pkt framework libs which use other libs from above
'port', 'table', 'pipeline',
# flow_classify lib depends on pkt framework table lib
- 'flow_classify', 'bpf', 'graph', 'telemetry']
+ 'flow_classify', 'bpf', 'graph', 'node', 'telemetry']
if is_windows
libraries = ['kvargs','eal'] # only supported libraries for windows
@@ -186,6 +186,9 @@ foreach l:libraries
dpdk_libraries = [shared_lib] + dpdk_libraries
dpdk_static_libraries = [static_lib] + dpdk_static_libraries
+ if libname == 'rte_node'
+ dpdk_graph_nodes = [static_lib]
+ endif
endif # sources.length() > 0
set_variable('shared_rte_' + name, shared_dep)
diff --git a/meson.build b/meson.build
index b7ae9c8d9..811c96421 100644
--- a/meson.build
+++ b/meson.build
@@ -16,6 +16,7 @@ cc = meson.get_compiler('c')
dpdk_conf = configuration_data()
dpdk_libraries = []
dpdk_static_libraries = []
+dpdk_graph_nodes = []
dpdk_driver_classes = []
dpdk_drivers = []
dpdk_extra_ldflags = []
diff --git a/mk/rte.app.mk b/mk/rte.app.mk
index b1195f09a..68d7806a4 100644
--- a/mk/rte.app.mk
+++ b/mk/rte.app.mk
@@ -99,6 +99,7 @@ _LDLIBS-$(CONFIG_RTE_LIBRTE_REORDER) += -lrte_reorder
_LDLIBS-$(CONFIG_RTE_LIBRTE_SCHED) += -lrte_sched
_LDLIBS-$(CONFIG_RTE_LIBRTE_RCU) += -lrte_rcu
_LDLIBS-$(CONFIG_RTE_LIBRTE_GRAPH) += -lrte_graph
+_LDLIBS-$(CONFIG_RTE_LIBRTE_NODE) += -lrte_node
ifeq ($(CONFIG_RTE_EXEC_ENV_LINUX),y)
_LDLIBS-$(CONFIG_RTE_LIBRTE_KNI) += -lrte_kni
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 16/28] node: add ethdev Rx node
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (14 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 15/28] node: add log infra and null node jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 17/28] node: add ethdev Tx node jerinj
` (13 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add source rte_node ethdev_rx process function and register
it. This node is a source node that will be called periodically
and when called, performs rte_eth_rx_burst() on a specific
(port, queue) pair and enqueue them as stream of objects to
next node.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/Makefile | 3 +-
lib/librte_node/ethdev_rx.c | 221 +++++++++++++++++++++++++++++++
lib/librte_node/ethdev_rx_priv.h | 81 +++++++++++
lib/librte_node/meson.build | 4 +-
4 files changed, 306 insertions(+), 3 deletions(-)
create mode 100644 lib/librte_node/ethdev_rx.c
create mode 100644 lib/librte_node/ethdev_rx_priv.h
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index dbc8e1d44..314149385 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -11,12 +11,13 @@ CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS)
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
CFLAGS += -fno-strict-aliasing
-LDLIBS += -lrte_eal -lrte_graph
+LDLIBS += -lrte_eal -lrte_graph -lrte_ethdev
EXPORT_MAP := rte_node_version.map
# all source are stored in SRCS-y
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/ethdev_rx.c b/lib/librte_node/ethdev_rx.c
new file mode 100644
index 000000000..5cc736598
--- /dev/null
+++ b/lib/librte_node/ethdev_rx.c
@@ -0,0 +1,221 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_ether.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_mbuf.h>
+
+#include "ethdev_rx_priv.h"
+#include "node_private.h"
+
+static struct ethdev_rx_node_main ethdev_rx_main;
+
+static __rte_always_inline uint16_t
+ethdev_rx_node_process_inline(struct rte_graph *graph, struct rte_node *node,
+ uint16_t port, uint16_t queue)
+{
+ uint16_t count, next_index = ETHDEV_RX_NEXT_IP4_LOOKUP;
+
+ /* Get pkts from port */
+ count = rte_eth_rx_burst(port, queue, (struct rte_mbuf **)node->objs,
+ RTE_GRAPH_BURST_SIZE);
+
+ if (!count)
+ return 0;
+ node->idx = count;
+ /* Enqueue to next node */
+ rte_node_next_stream_move(graph, node, next_index);
+
+ return count;
+}
+
+static __rte_always_inline uint16_t
+ethdev_rx_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t cnt)
+{
+ ethdev_rx_node_ctx_t *ctx = (ethdev_rx_node_ctx_t *)node->ctx;
+ uint16_t n_pkts = 0;
+
+ RTE_SET_USED(objs);
+ RTE_SET_USED(cnt);
+
+ n_pkts = ethdev_rx_node_process_inline(graph, node, ctx->port_id,
+ ctx->queue_id);
+ return n_pkts;
+}
+
+static inline uint32_t
+l3_ptype(uint16_t etype, uint32_t ptype)
+{
+ ptype = ptype & ~RTE_PTYPE_L3_MASK;
+ if (etype == rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV4))
+ ptype |= RTE_PTYPE_L3_IPV4_EXT_UNKNOWN;
+ else if (etype == rte_cpu_to_be_16(RTE_ETHER_TYPE_IPV6))
+ ptype |= RTE_PTYPE_L3_IPV6_EXT_UNKNOWN;
+ return ptype;
+}
+
+/* Callback for soft ptype parsing */
+static uint16_t
+eth_pkt_parse_cb(uint16_t port, uint16_t queue, struct rte_mbuf **mbufs,
+ uint16_t nb_pkts, uint16_t max_pkts, void *user_param)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3;
+ struct rte_ether_hdr *eth_hdr;
+ uint16_t etype, n_left;
+ struct rte_mbuf **pkts;
+
+ RTE_SET_USED(port);
+ RTE_SET_USED(queue);
+ RTE_SET_USED(max_pkts);
+ RTE_SET_USED(user_param);
+
+ pkts = mbufs;
+ n_left = nb_pkts;
+ while (n_left >= 12) {
+
+ /* Prefetch next-next mbufs */
+ rte_prefetch0(pkts[8]);
+ rte_prefetch0(pkts[9]);
+ rte_prefetch0(pkts[10]);
+ rte_prefetch0(pkts[11]);
+
+ /* Prefetch next mbuf data */
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[4], struct rte_ether_hdr *));
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[5], struct rte_ether_hdr *));
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[6], struct rte_ether_hdr *));
+ rte_prefetch0(
+ rte_pktmbuf_mtod(pkts[7], struct rte_ether_hdr *));
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+ pkts += 4;
+ n_left -= 4;
+
+ /* Extract ptype of mbuf0 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf0->packet_type = l3_ptype(etype, 0);
+
+ /* Extract ptype of mbuf1 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf1, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf1->packet_type = l3_ptype(etype, 0);
+
+ /* Extract ptype of mbuf2 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf2, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf2->packet_type = l3_ptype(etype, 0);
+
+ /* Extract ptype of mbuf3 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf3, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf3->packet_type = l3_ptype(etype, 0);
+ }
+
+ while (n_left > 0) {
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left -= 1;
+
+ /* Extract ptype of mbuf0 */
+ eth_hdr = rte_pktmbuf_mtod(mbuf0, struct rte_ether_hdr *);
+ etype = eth_hdr->ether_type;
+ mbuf0->packet_type = l3_ptype(etype, 0);
+ }
+
+ return nb_pkts;
+}
+
+#define MAX_PTYPES 16
+static int
+ethdev_ptype_setup(uint16_t port, uint16_t queue)
+{
+ uint8_t l3_ipv4 = 0, l3_ipv6 = 0;
+ uint32_t ptypes[MAX_PTYPES];
+ int i, rc;
+
+ /* Check IPv4 & IPv6 ptype support */
+ rc = rte_eth_dev_get_supported_ptypes(port, RTE_PTYPE_L3_MASK, ptypes,
+ MAX_PTYPES);
+ for (i = 0; i < rc; i++) {
+ if (ptypes[i] & RTE_PTYPE_L3_IPV4)
+ l3_ipv4 = 1;
+ if (ptypes[i] & RTE_PTYPE_L3_IPV6)
+ l3_ipv6 = 1;
+ }
+
+ if (!l3_ipv4 || !l3_ipv6) {
+ node_info("ethdev_rx",
+ "Enabling ptype callback for required ptypes on port %u\n",
+ port);
+
+ if (!rte_eth_add_rx_callback(port, queue, eth_pkt_parse_cb,
+ NULL)) {
+ node_err("ethdev_rx",
+ "Failed to add rx ptype cb: port=%d, queue=%d\n",
+ port, queue);
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
+static int
+ethdev_rx_node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ ethdev_rx_node_ctx_t *ctx = (ethdev_rx_node_ctx_t *)node->ctx;
+ ethdev_rx_node_elem_t *elem = ethdev_rx_main.head;
+
+ RTE_SET_USED(graph);
+
+ while (elem) {
+ if (elem->nid == node->id) {
+ /* Update node specific context */
+ memcpy(ctx, &elem->ctx, sizeof(ethdev_rx_node_ctx_t));
+ break;
+ }
+ elem = elem->next;
+ }
+
+ RTE_VERIFY(elem != NULL);
+
+ /* Check and setup ptype */
+ return ethdev_ptype_setup(ctx->port_id, ctx->queue_id);
+}
+
+struct ethdev_rx_node_main *
+ethdev_rx_get_node_data_get(void)
+{
+ return ðdev_rx_main;
+}
+
+static struct rte_node_register ethdev_rx_node_base = {
+ .process = ethdev_rx_node_process,
+ .flags = RTE_NODE_SOURCE_F,
+ .name = "ethdev_rx",
+
+ .init = ethdev_rx_node_init,
+
+ .nb_edges = ETHDEV_RX_NEXT_MAX,
+ .next_nodes = {[ETHDEV_RX_NEXT_IP4_LOOKUP] = "ip4_lookup"},
+};
+
+struct rte_node_register *
+ethdev_rx_node_get(void)
+{
+ return ðdev_rx_node_base;
+}
+
+RTE_NODE_REGISTER(ethdev_rx_node_base);
diff --git a/lib/librte_node/ethdev_rx_priv.h b/lib/librte_node/ethdev_rx_priv.h
new file mode 100644
index 000000000..2d7195a36
--- /dev/null
+++ b/lib/librte_node/ethdev_rx_priv.h
@@ -0,0 +1,81 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#ifndef __INCLUDE_ETHDEV_RX_PRIV_H__
+#define __INCLUDE_ETHDEV_RX_PRIV_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+struct ethdev_rx_node_elem;
+struct ethdev_rx_node_ctx;
+typedef struct ethdev_rx_node_elem ethdev_rx_node_elem_t;
+typedef struct ethdev_rx_node_ctx ethdev_rx_node_ctx_t;
+
+/**
+ * @internal
+ *
+ * Ethernet device Rx node context structure.
+ */
+struct ethdev_rx_node_ctx {
+ uint16_t port_id; /**< Port identifier of the Rx node. */
+ uint16_t queue_id; /**< Queue identifier of the Rx node. */
+};
+
+/**
+ * @internal
+ *
+ * Ethernet device Rx node list element structure.
+ */
+struct ethdev_rx_node_elem {
+ struct ethdev_rx_node_elem *next;
+ /**< Pointer to the next Rx node element. */
+ struct ethdev_rx_node_ctx ctx;
+ /**< Rx node context. */
+ rte_node_t nid;
+ /**< Node identifier of the Rx node. */
+};
+
+enum ethdev_rx_next_nodes {
+ ETHDEV_RX_NEXT_IP4_LOOKUP,
+ ETHDEV_RX_NEXT_MAX,
+};
+
+/**
+ * @internal
+ *
+ * Ethernet Rx node main structure.
+ */
+struct ethdev_rx_node_main {
+ ethdev_rx_node_elem_t *head;
+ /**< Pointer to the head Rx node element. */
+};
+
+/**
+ * @internal
+ *
+ * Get the Ethernet Rx node data.
+ *
+ * @return
+ * Pointer to Ethernet Rx node data.
+ */
+struct ethdev_rx_node_main *ethdev_rx_get_node_data_get(void);
+
+/**
+ * @internal
+ *
+ * Get the Ethernet Rx node.
+ *
+ * @retrun
+ * Pointer to the Ethernet Rx node.
+ */
+struct rte_node_register *ethdev_rx_node_get(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_ETHDEV_RX_PRIV_H__ */
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index a97813ad4..94caa6c23 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -1,8 +1,8 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(C) 2020 Marvell International Ltd.
-sources = files('null.c', 'log.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c')
allow_experimental_apis = true
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
cflags += '-fno-strict-aliasing'
-deps += ['graph']
+deps += ['graph', 'ethdev']
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 17/28] node: add ethdev Tx node
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (15 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 16/28] node: add ethdev Rx node jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 18/28] node: add ethdev Rx and Tx node ctrl API jerinj
` (12 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add rte_node ethdev_tx process function and register it to
graph infra. This node has a specific (port, tx-queue) as context
and it enqueue's all the packets received to that specific queue pair.
When rte_eth_tx_burst() i.e enqueue to queue pair fails, packets
are forwarded to pkt_drop node to be free'd.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/Makefile | 3 +-
lib/librte_node/ethdev_tx.c | 86 ++++++++++++++++++++++++++++++++
lib/librte_node/ethdev_tx_priv.h | 62 +++++++++++++++++++++++
lib/librte_node/meson.build | 4 +-
4 files changed, 152 insertions(+), 3 deletions(-)
create mode 100644 lib/librte_node/ethdev_tx.c
create mode 100644 lib/librte_node/ethdev_tx_priv.h
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index 314149385..7428f6c43 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -11,7 +11,7 @@ CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS)
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
CFLAGS += -fno-strict-aliasing
-LDLIBS += -lrte_eal -lrte_graph -lrte_ethdev
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev
EXPORT_MAP := rte_node_version.map
@@ -19,5 +19,6 @@ EXPORT_MAP := rte_node_version.map
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/ethdev_tx.c b/lib/librte_node/ethdev_tx.c
new file mode 100644
index 000000000..075149089
--- /dev/null
+++ b/lib/librte_node/ethdev_tx.c
@@ -0,0 +1,86 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_mbuf.h>
+
+#include "ethdev_tx_priv.h"
+
+static struct ethdev_tx_node_main ethdev_tx_main;
+
+static uint16_t
+ethdev_tx_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ ethdev_tx_node_ctx_t *ctx = (ethdev_tx_node_ctx_t *)node->ctx;
+ uint16_t port, queue;
+ uint16_t count;
+
+ /* Get Tx port id */
+ port = ctx->port;
+ queue = ctx->queue;
+
+ count = rte_eth_tx_burst(port, queue, (struct rte_mbuf **)objs,
+ nb_objs);
+
+ /* Redirect unsent pkts to drop node */
+ if (count != nb_objs) {
+ rte_node_enqueue(graph, node, ETHDEV_TX_NEXT_PKT_DROP,
+ &objs[count], nb_objs - count);
+ }
+
+ return count;
+}
+
+static int
+ethdev_tx_node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ ethdev_tx_node_ctx_t *ctx = (ethdev_tx_node_ctx_t *)node->ctx;
+ uint64_t port_id = RTE_MAX_ETHPORTS;
+ int i;
+
+ /* Find our port id */
+ for (i = 0; i < RTE_MAX_ETHPORTS; i++) {
+ if (ethdev_tx_main.nodes[i] == node->id) {
+ port_id = i;
+ break;
+ }
+ }
+ RTE_VERIFY(port_id < RTE_MAX_ETHPORTS);
+
+ /* Update port and queue */
+ ctx->port = port_id;
+ ctx->queue = graph->id;
+
+ return 0;
+}
+
+struct ethdev_tx_node_main *
+ethdev_tx_node_data_get(void)
+{
+ return ðdev_tx_main;
+}
+
+static struct rte_node_register ethdev_tx_node_base = {
+ .process = ethdev_tx_node_process,
+ .name = "ethdev_tx",
+
+ .init = ethdev_tx_node_init,
+
+ .nb_edges = ETHDEV_TX_NEXT_MAX,
+ .next_nodes = {
+ [ETHDEV_TX_NEXT_PKT_DROP] = "pkt_drop",
+ },
+};
+
+struct rte_node_register *
+ethdev_tx_node_get(void)
+{
+ return ðdev_tx_node_base;
+}
+
+RTE_NODE_REGISTER(ethdev_tx_node_base);
diff --git a/lib/librte_node/ethdev_tx_priv.h b/lib/librte_node/ethdev_tx_priv.h
new file mode 100644
index 000000000..586bff44a
--- /dev/null
+++ b/lib/librte_node/ethdev_tx_priv.h
@@ -0,0 +1,62 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#ifndef __INCLUDE_ETHDEV_TX_PRIV_H__
+#define __INCLUDE_ETHDEV_TX_PRIV_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+struct ethdev_tx_node_ctx;
+typedef struct ethdev_tx_node_ctx ethdev_tx_node_ctx_t;
+
+enum ethdev_tx_next_nodes {
+ ETHDEV_TX_NEXT_PKT_DROP,
+ ETHDEV_TX_NEXT_MAX,
+};
+
+/**
+ * @internal
+ *
+ * Ethernet Tx node context structure.
+ */
+struct ethdev_tx_node_ctx {
+ uint16_t port; /**< Port identifier of the Ethernet Tx node. */
+ uint16_t queue; /**< Queue identifier of the Ethernet Tx node. */
+};
+
+/**
+ * @internal
+ *
+ * Ethernet Tx node main structure.
+ */
+struct ethdev_tx_node_main {
+ uint32_t nodes[RTE_MAX_ETHPORTS]; /**< Tx nodes for each ethdev port. */
+};
+
+/**
+ * @internal
+ *
+ * Get the Ethernet Tx node data.
+ *
+ * @return
+ * Pointer to Ethernet Tx node data.
+ */
+struct ethdev_tx_node_main *ethdev_tx_node_data_get(void);
+
+/**
+ * @internal
+ *
+ * Get the Ethernet Tx node.
+ *
+ * @retrun
+ * Pointer to the Ethernet Tx node.
+ */
+struct rte_node_register *ethdev_tx_node_get(void);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_ETHDEV_TX_PRIV_H__ */
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index 94caa6c23..505c76abd 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -1,8 +1,8 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(C) 2020 Marvell International Ltd.
-sources = files('null.c', 'log.c', 'ethdev_rx.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c')
allow_experimental_apis = true
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
cflags += '-fno-strict-aliasing'
-deps += ['graph', 'ethdev']
+deps += ['graph', 'mbuf', 'ethdev']
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 18/28] node: add ethdev Rx and Tx node ctrl API
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (16 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 17/28] node: add ethdev Tx node jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 19/28] node: ipv4 lookup for arm64 jerinj
` (11 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: John McNamara, Marko Kovacevic, Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add ctrl api to setup ethdev_rx and ethdev_tx node.
This ctrl api clones 'N' number of ethdev_rx and ethdev_tx
nodes with specific (port, queue) pairs updated in their context.
All the ethdev ports and queues are setup before this api
is called.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
doc/api/doxy-api-index.md | 2 +
lib/librte_node/Makefile | 6 +-
lib/librte_node/ethdev_ctrl.c | 100 +++++++++++++++++++++++++++
lib/librte_node/meson.build | 5 +-
lib/librte_node/node_private.h | 74 ++++++++++++++++++++
lib/librte_node/rte_node_eth_api.h | 70 +++++++++++++++++++
lib/librte_node/rte_node_version.map | 1 +
7 files changed, 255 insertions(+), 3 deletions(-)
create mode 100644 lib/librte_node/ethdev_ctrl.c
create mode 100644 lib/librte_node/rte_node_eth_api.h
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index fd2ff64d7..1784674df 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -161,6 +161,8 @@ The public API headers are grouped by topics:
[table_action] (@ref rte_table_action.h)
* [graph] (@ref rte_graph.h):
[graph_worker] (@ref rte_graph_worker.h)
+ * graph_nodes:
+ [eth_node] (@ref rte_node_eth_api.h),
- **basic**:
[approx fraction] (@ref rte_approx.h),
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index 7428f6c43..ea5fa77f7 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -11,7 +11,7 @@ CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS)
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
CFLAGS += -fno-strict-aliasing
-LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev -lrte_mempool
EXPORT_MAP := rte_node_version.map
@@ -20,5 +20,9 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += null.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
+
+# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_eth_api.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/ethdev_ctrl.c b/lib/librte_node/ethdev_ctrl.c
new file mode 100644
index 000000000..b2ac5e2c4
--- /dev/null
+++ b/lib/librte_node/ethdev_ctrl.c
@@ -0,0 +1,100 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_ether.h>
+#include <rte_graph.h>
+
+#include "rte_node_eth_api.h"
+
+#include "ethdev_rx_priv.h"
+#include "ethdev_tx_priv.h"
+#include "node_private.h"
+
+static struct ethdev_ctrl {
+ uint16_t nb_graphs;
+} ctrl;
+
+int
+rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
+ uint16_t nb_graphs)
+{
+ struct ethdev_tx_node_main *tx_node_data;
+ uint16_t tx_q_used, rx_q_used, port_id;
+ struct rte_node_register *tx_node;
+ char name[RTE_NODE_NAMESIZE];
+ struct rte_mempool *mp;
+ uint32_t id;
+ int i, j;
+
+ tx_node_data = ethdev_tx_node_data_get();
+ tx_node = ethdev_tx_node_get();
+ for (i = 0; i < nb_confs; i++) {
+ port_id = conf[i].port_id;
+
+ if (!rte_eth_dev_is_valid_port(port_id))
+ return -EINVAL;
+
+ /* Check for mbuf minimum private size requirement */
+ for (j = 0; j < conf[i].mp_count; j++) {
+ mp = conf[i].mp[j];
+ if (!mp)
+ continue;
+ /* Check for minimum private space */
+ if (rte_pktmbuf_priv_size(mp) <
+ RTE_NODE_MBUF_PRIV2_SIZE) {
+ node_err("ethdev",
+ "Minimum mbuf priv size requirement not met by mp %s",
+ mp->name);
+ return -EINVAL;
+ }
+ }
+
+ rx_q_used = conf[i].num_rx_queues;
+ tx_q_used = conf[i].num_tx_queues;
+ /* Check if we have a txq for each worker */
+ if (tx_q_used < nb_graphs)
+ return -EINVAL;
+
+ /* Create node for each rx port queue pair */
+ for (j = 0; j < rx_q_used; j++) {
+ struct ethdev_rx_node_main *rx_node_data;
+ struct rte_node_register *rx_node;
+ ethdev_rx_node_elem_t *elem;
+
+ rx_node_data = ethdev_rx_get_node_data_get();
+ rx_node = ethdev_rx_node_get();
+ snprintf(name, sizeof(name), "%u-%u", port_id, j);
+ /* Clone a new rx node with same edges as parent */
+ id = rte_node_clone(rx_node->id, name);
+ if (id == RTE_NODE_ID_INVALID)
+ return -EIO;
+
+ /* Add it to list of ethdev rx nodes for lookup */
+ elem = malloc(sizeof(ethdev_rx_node_elem_t));
+ memset(elem, 0, sizeof(ethdev_rx_node_elem_t));
+ elem->ctx.port_id = port_id;
+ elem->ctx.queue_id = j;
+ elem->nid = id;
+ elem->next = rx_node_data->head;
+ rx_node_data->head = elem;
+
+ node_dbg("ethdev", "Rx node %s-%s: is at %u",
+ rx_node->name, name, id);
+ }
+
+ /* Create a per port tx node from base node */
+ snprintf(name, sizeof(name), "%u", port_id);
+ /* Clone a new node with same edges as parent */
+ id = rte_node_clone(tx_node->id, name);
+ tx_node_data->nodes[port_id] = id;
+
+ node_dbg("ethdev", "Tx node %s-%s: is at %u", tx_node->name,
+ name, id);
+ }
+
+ ctrl.nb_graphs = nb_graphs;
+ return 0;
+}
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index 505c76abd..af2eb2d91 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -1,8 +1,9 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(C) 2020 Marvell International Ltd.
-sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ethdev_ctrl.c')
+headers = files('rte_node_eth_api.h')
allow_experimental_apis = true
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
cflags += '-fno-strict-aliasing'
-deps += ['graph', 'mbuf', 'ethdev']
+deps += ['graph', 'mbuf', 'lpm', 'ethdev', 'mempool', 'cryptodev']
diff --git a/lib/librte_node/node_private.h b/lib/librte_node/node_private.h
index f30902a94..141448546 100644
--- a/lib/librte_node/node_private.h
+++ b/lib/librte_node/node_private.h
@@ -6,7 +6,9 @@
#define __NODE_PRIVATE_H__
#include <rte_common.h>
+#include <rte_crypto.h>
#include <rte_log.h>
+#include <rte_mbuf.h>
extern int rte_node_logtype;
#define NODE_LOG(level, node_name, ...) \
@@ -19,4 +21,76 @@ extern int rte_node_logtype;
#define node_info(node_name, ...) NODE_LOG(INFO, node_name, __VA_ARGS__)
#define node_dbg(node_name, ...) NODE_LOG(DEBUG, node_name, __VA_ARGS__)
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node mbuf private data to store next hop, ttl and checksum.
+ */
+struct rte_node_mbuf_priv1 {
+ union {
+ /* IP4 rewrite */
+ struct {
+ uint16_t nh;
+ uint16_t ttl;
+ uint32_t cksum;
+ };
+
+ uint64_t u;
+ };
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Node mbuf private data to store crypto operation.
+ */
+struct rte_node_mbuf_priv2 {
+ union {
+ /* Sym crypto */
+ struct {
+ struct rte_crypto_op op;
+ };
+ };
+} __rte_cache_aligned;
+
+#define RTE_NODE_MBUF_PRIV2_SIZE sizeof(struct rte_node_mbuf_priv2)
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get mbuf_priv1 pointer from rte_mbuf.
+ *
+ * @param
+ * Pointer to the rte_mbuf.
+ *
+ * @return
+ * Pointer to the mbuf_priv1.
+ */
+static __rte_always_inline struct rte_node_mbuf_priv1 *
+rte_node_mbuf_priv1(struct rte_mbuf *m)
+{
+ return (struct rte_node_mbuf_priv1 *)&m->udata64;
+}
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Get mbuf_priv2 pointer from rte_mbuf.
+ *
+ * @param
+ * Pointer to the rte_mbuf.
+ *
+ * @return
+ * Pointer to the mbuf_priv2.
+ */
+static __rte_always_inline struct rte_node_mbuf_priv2 *
+rte_node_mbuf_priv2(struct rte_mbuf *m)
+{
+ return (struct rte_node_mbuf_priv2 *)rte_mbuf_to_priv(m);
+}
+
#endif /* __NODE_PRIVATE_H__ */
diff --git a/lib/librte_node/rte_node_eth_api.h b/lib/librte_node/rte_node_eth_api.h
new file mode 100644
index 000000000..39b31b45b
--- /dev/null
+++ b/lib/librte_node/rte_node_eth_api.h
@@ -0,0 +1,70 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __INCLUDE_RTE_NODE_ETH_API_H__
+#define __INCLUDE_RTE_NODE_ETH_API_H__
+
+/**
+ * @file rte_node_eth_api.h
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * This API allows to setup ethdev_rx and ethdev_tx nodes
+ * and its queue associations.
+ *
+ */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+#include <rte_mempool.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Port config for ethdev_rx and ethdev_tx node.
+ */
+struct rte_node_ethdev_config {
+ uint16_t port_id;
+ /**< Port identifier */
+ uint16_t num_rx_queues;
+ /**< Number of Rx queues. */
+ uint16_t num_tx_queues;
+ /**< Number of Tx queues. */
+ struct rte_mempool **mp;
+ /**< Array of mempools associated to Rx queue. */
+ uint16_t mp_count;
+ /**< Size of mp array. */
+};
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Initializes ethdev nodes.
+ *
+ * @param cfg
+ * Array of ethdev config that identifies which port's
+ * ethdev_rx and ethdev_tx nodes need to be created
+ * and queue association.
+ * @param cnt
+ * Size of cfg array.
+ * @param nb_graphs
+ * Number of graphs that will be used.
+ *
+ * @return
+ * 0 on successful initialization, negative otherwise.
+ */
+__rte_experimental
+int rte_node_eth_config(struct rte_node_ethdev_config *cfg,
+ uint16_t cnt, uint16_t nb_graphs);
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_RTE_NODE_ETH_API_H__ */
diff --git a/lib/librte_node/rte_node_version.map b/lib/librte_node/rte_node_version.map
index f87163bb9..c6c71bd02 100644
--- a/lib/librte_node/rte_node_version.map
+++ b/lib/librte_node/rte_node_version.map
@@ -1,6 +1,7 @@
EXPERIMENTAL {
global:
+ rte_node_eth_config;
rte_node_logtype;
local: *;
};
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 19/28] node: ipv4 lookup for arm64
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (17 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 18/28] node: add ethdev Rx and Tx node ctrl API jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 20/28] node: ipv4 lookup for x86 jerinj
` (10 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: John McNamara, Marko Kovacevic, Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add arm64 specific IPv4 lookup process function
for ip4_lookup node. This node performs LPM lookup
on every packet received and forwards it to a next
node that is identified by lookup result.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
doc/api/doxy-api-index.md | 1 +
lib/librte_node/Makefile | 4 +-
lib/librte_node/ip4_lookup.c | 301 +++++++++++++++++++++++++++++
lib/librte_node/meson.build | 5 +-
lib/librte_node/rte_node_ip4_api.h | 43 +++++
5 files changed, 351 insertions(+), 3 deletions(-)
create mode 100644 lib/librte_node/ip4_lookup.c
create mode 100644 lib/librte_node/rte_node_ip4_api.h
diff --git a/doc/api/doxy-api-index.md b/doc/api/doxy-api-index.md
index 1784674df..3908fddde 100644
--- a/doc/api/doxy-api-index.md
+++ b/doc/api/doxy-api-index.md
@@ -163,6 +163,7 @@ The public API headers are grouped by topics:
[graph_worker] (@ref rte_graph_worker.h)
* graph_nodes:
[eth_node] (@ref rte_node_eth_api.h),
+ [ip4_node] (@ref rte_node_ip4_api.h)
- **basic**:
[approx fraction] (@ref rte_approx.h),
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index ea5fa77f7..ad3f2e349 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -11,7 +11,7 @@ CFLAGS += -O3 -DALLOW_EXPERIMENTAL_API
CFLAGS += $(WERROR_FLAGS)
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
CFLAGS += -fno-strict-aliasing
-LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_ethdev -lrte_mempool
+LDLIBS += -lrte_eal -lrte_graph -lrte_mbuf -lrte_lpm -lrte_ethdev -lrte_mempool
EXPORT_MAP := rte_node_version.map
@@ -21,8 +21,10 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += log.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
# install header files
+SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_eth_api.h
include $(RTE_SDK)/mk/rte.lib.mk
diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c
new file mode 100644
index 000000000..dddabc6c2
--- /dev/null
+++ b/lib/librte_node/ip4_lookup.c
@@ -0,0 +1,301 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <arpa/inet.h>
+#include <sys/socket.h>
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_ether.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_ip.h>
+#include <rte_lpm.h>
+#include <rte_mbuf.h>
+#include <rte_tcp.h>
+#include <rte_udp.h>
+
+#include "rte_node_ip4_api.h"
+
+#include "node_private.h"
+
+#define IPV4_L3FWD_LPM_MAX_RULES 1024
+#define IPV4_L3FWD_LPM_NUMBER_TBL8S (1 << 8)
+
+/* IP4 Lookup global data struct */
+struct ip4_lookup_node_main {
+ struct rte_lpm *lpm_tbl[RTE_MAX_NUMA_NODES];
+};
+
+#if defined(RTE_MACHINE_CPUFLAG_NEON)
+/* ARM64 NEON */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+ struct rte_ipv4_hdr *ipv4_hdr;
+ void **to_next, **from;
+ uint16_t last_spec = 0;
+ rte_edge_t next_index;
+ uint16_t n_left_from;
+ struct rte_lpm *lpm;
+ uint16_t held = 0;
+ uint32_t drop_nh;
+ rte_xmm_t result;
+ rte_xmm_t priv01;
+ rte_xmm_t priv23;
+ int32x4_t dip;
+ int rc, i;
+
+ /* Speculative next */
+ next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+ /* Drop node */
+ drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
+
+ /* Get socket specific LPM from ctx */
+ lpm = *((struct rte_lpm **)node->ctx);
+
+ pkts = (struct rte_mbuf **)objs;
+ from = objs;
+ n_left_from = nb_objs;
+
+#define OBJS_PER_CLINE (RTE_CACHE_LINE_SIZE / sizeof(void *))
+ for (i = OBJS_PER_CLINE; i < RTE_GRAPH_BURST_SIZE; i += OBJS_PER_CLINE)
+ rte_prefetch0(&objs[i]);
+
+ for (i = 0; i < 4 && i < n_left_from; i++)
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[i], void *,
+ sizeof(struct rte_ether_hdr)));
+
+ /* Get stream for the speculated next node */
+ to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+ while (n_left_from >= 4) {
+#if RTE_GRAPH_BURST_SIZE > 64
+ /* Prefetch next-next mbufs */
+ if (likely(n_left_from > 11)) {
+ rte_prefetch0(pkts[8]);
+ rte_prefetch0(pkts[9]);
+ rte_prefetch0(pkts[10]);
+ rte_prefetch0(pkts[11]);
+ }
+#endif
+ /* Prefetch next mbuf data */
+ if (likely(n_left_from > 7)) {
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[4], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[5], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[6], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[7], void *,
+ sizeof(struct rte_ether_hdr)));
+ }
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+
+ pkts += 4;
+ n_left_from -= 4;
+
+ /* Extract DIP of mbuf0 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 0);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv01.u16[1] = ipv4_hdr->time_to_live;
+ priv01.u32[1] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf1 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf1, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 1);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv01.u16[5] = ipv4_hdr->time_to_live;
+ priv01.u32[3] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf2 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf2, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 2);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv23.u16[1] = ipv4_hdr->time_to_live;
+ priv23.u32[1] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf3 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf3, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 3);
+
+ dip = vreinterpretq_s32_u8(
+ vrev32q_u8(vreinterpretq_u8_s32(dip)));
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv23.u16[5] = ipv4_hdr->time_to_live;
+ priv23.u32[3] = ipv4_hdr->hdr_checksum;
+
+ /* Perform LPM lookup to get NH and next node */
+ rte_lpm_lookupx4(lpm, dip, result.u32, drop_nh);
+ priv01.u16[0] = result.u16[0];
+ priv01.u16[4] = result.u16[2];
+ priv23.u16[0] = result.u16[4];
+ priv23.u16[4] = result.u16[6];
+
+ rte_node_mbuf_priv1(mbuf0)->u = priv01.u64[0];
+ rte_node_mbuf_priv1(mbuf1)->u = priv01.u64[1];
+ rte_node_mbuf_priv1(mbuf2)->u = priv23.u64[0];
+ rte_node_mbuf_priv1(mbuf3)->u = priv23.u64[1];
+
+ /* Enqueue four to next node */
+ rte_edge_t fix_spec = ((next_index == result.u16[1]) &&
+ (result.u16[1] == result.u16[3]) &&
+ (result.u16[3] == result.u16[5]) &&
+ (result.u16[5] == result.u16[7]));
+
+ if (unlikely(fix_spec == 0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ /* Next0 */
+ if (next_index == result.u16[1]) {
+ to_next[0] = from[0];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[1],
+ from[0]);
+ }
+
+ /* Next1 */
+ if (next_index == result.u16[3]) {
+ to_next[0] = from[1];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[3],
+ from[1]);
+ }
+
+ /* Next2 */
+ if (next_index == result.u16[5]) {
+ to_next[0] = from[2];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[5],
+ from[2]);
+ }
+
+ /* Next3 */
+ if (next_index == result.u16[7]) {
+ to_next[0] = from[3];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[7],
+ from[3]);
+ }
+
+ from += 4;
+ } else {
+ last_spec += 4;
+ }
+ }
+
+ while (n_left_from > 0) {
+ uint32_t next_hop;
+ uint16_t next0;
+
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left_from -= 1;
+
+ /* Extract DIP of mbuf0 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+ rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
+ &next_hop);
+ next_hop = (rc == 0) ? next_hop : drop_nh;
+
+ rte_node_mbuf_priv1(mbuf0)->nh = (uint16_t)next_hop;
+ next_hop = next_hop >> 16;
+ next0 = (uint16_t)next_hop;
+
+ if (unlikely(next_index ^ next0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ rte_node_enqueue_x1(graph, node, next0, from[0]);
+ from += 1;
+ } else {
+ last_spec += 1;
+ }
+ }
+
+ /* !!! Home run !!! */
+ if (likely(last_spec == nb_objs)) {
+ rte_node_next_stream_move(graph, node, next_index);
+ return nb_objs;
+ }
+ held += last_spec;
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ rte_node_next_stream_put(graph, node, next_index, held);
+
+ return nb_objs;
+}
+
+#else
+
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ RTE_SET_USED(graph);
+ RTE_SET_USED(node);
+ RTE_SET_USED(objs);
+ RTE_SET_USED(nb_objs);
+ return nb_objs;
+}
+
+#endif
+
+static int
+ip4_lookup_node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+ RTE_SET_USED(graph);
+ RTE_SET_USED(node);
+
+ node_dbg("ip4_lookup", "Initialized ip4_lookup node");
+
+ return 0;
+}
+
+static struct rte_node_register ip4_lookup_node = {
+ .process = ip4_lookup_node_process,
+ .name = "ip4_lookup",
+
+ .init = ip4_lookup_node_init,
+
+ .nb_edges = RTE_NODE_IP4_LOOKUP_NEXT_MAX,
+ .next_nodes = {
+ [RTE_NODE_IP4_LOOKUP_NEXT_REWRITE] = "ip4_rewrite",
+ [RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP] = "pkt_drop",
+ },
+};
+
+RTE_NODE_REGISTER(ip4_lookup_node);
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index af2eb2d91..702d21a24 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -1,8 +1,9 @@
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(C) 2020 Marvell International Ltd.
-sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ethdev_ctrl.c')
-headers = files('rte_node_eth_api.h')
+sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
+ 'ethdev_ctrl.c')
+headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
allow_experimental_apis = true
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
cflags += '-fno-strict-aliasing'
diff --git a/lib/librte_node/rte_node_ip4_api.h b/lib/librte_node/rte_node_ip4_api.h
new file mode 100644
index 000000000..37c12bf82
--- /dev/null
+++ b/lib/librte_node/rte_node_ip4_api.h
@@ -0,0 +1,43 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __INCLUDE_RTE_NODE_IP4_API_H__
+#define __INCLUDE_RTE_NODE_IP4_API_H__
+
+/**
+ * @file rte_node_ip4_api.h
+ *
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * This API allows to do control path functions of ip4_* nodes
+ * like ip4_lookup, ip4_rewrite.
+ *
+ */
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * IP4 lookup next nodes.
+ */
+enum rte_node_ip4_lookup_next {
+ RTE_NODE_IP4_LOOKUP_NEXT_REWRITE,
+ /**< Rewrite node. */
+ RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP,
+ /**< Packet drop node. */
+ RTE_NODE_IP4_LOOKUP_NEXT_MAX,
+ /**< Number of next nodes of lookup node. */
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_RTE_NODE_IP4_API_H__ */
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 20/28] node: ipv4 lookup for x86
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (18 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 19/28] node: ipv4 lookup for arm64 jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-27 8:40 ` Jerin Jacob
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 21/28] node: add ipv4 rewrite node jerinj
` (9 subsequent siblings)
29 siblings, 1 reply; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Pavan Nikhilesh <pbhagavatula@marvell.com>
Add IPv4 lookup process function for ip4_lookup
rte_node. This node performs LPM lookup using x86_64
vector supported RTE_LPM API on every packet received
and forwards it to a next node that is identified by
lookup result.
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/ip4_lookup.c | 238 +++++++++++++++++++++++++++++++++++
1 file changed, 238 insertions(+)
diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c
index dddabc6c2..a9e204148 100644
--- a/lib/librte_node/ip4_lookup.c
+++ b/lib/librte_node/ip4_lookup.c
@@ -259,6 +259,244 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
return nb_objs;
}
+#elif defined(RTE_ARCH_X86)
+
+/* X86 SSE */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+ rte_edge_t next0, next1, next2, next3, next_index;
+ struct rte_ipv4_hdr *ipv4_hdr;
+ uint32_t ip0, ip1, ip2, ip3;
+ void **to_next, **from;
+ uint16_t last_spec = 0;
+ uint16_t n_left_from;
+ struct rte_lpm *lpm;
+ uint16_t held = 0;
+ uint32_t drop_nh;
+ rte_xmm_t dst;
+ __m128i dip; /* SSE register */
+ int rc, i;
+
+ /* Speculative next */
+ next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+ /* Drop node */
+ drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
+
+ /* Get socket specific LPM from ctx */
+ lpm = *((struct rte_lpm **)node->ctx);
+
+ pkts = (struct rte_mbuf **)objs;
+ from = objs;
+ n_left_from = nb_objs;
+
+ if (n_left_from >= 4) {
+ for (i = 0; i < 4; i++)
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[i], void *,
+ sizeof(struct rte_ether_hdr)));
+ }
+
+ /* Get stream for the speculated next node */
+ to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+ while (n_left_from >= 4) {
+ /* Prefetch next-next mbufs */
+ if (likely(n_left_from > 11)) {
+ rte_prefetch0(pkts[8]);
+ rte_prefetch0(pkts[9]);
+ rte_prefetch0(pkts[10]);
+ rte_prefetch0(pkts[11]);
+ }
+
+ /* Prefetch next mbuf data */
+ if (likely(n_left_from > 7)) {
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[4], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[5], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[6], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[7], void *,
+ sizeof(struct rte_ether_hdr)));
+ }
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+
+ pkts += 4;
+ n_left_from -= 4;
+
+ /* Extract DIP of mbuf0 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ ip0 = ipv4_hdr->dst_addr;
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+ /* Extract DIP of mbuf1 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf1, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ ip1 = ipv4_hdr->dst_addr;
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr->time_to_live;
+
+ /* Extract DIP of mbuf2 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf2, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ ip2 = ipv4_hdr->dst_addr;
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr->time_to_live;
+
+ /* Extract DIP of mbuf3 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf3, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ ip3 = ipv4_hdr->dst_addr;
+
+ /* Prepare for lookup x4 */
+ dip = _mm_set_epi32(ip3, ip2, ip1, ip0);
+
+ /* Byte swap 4 IPV4 addresses. */
+ const __m128i bswap_mask = _mm_set_epi8(
+ 12, 13, 14, 15, 8, 9, 10, 11, 4, 5, 6, 7, 0, 1, 2, 3);
+ dip = _mm_shuffle_epi8(dip, bswap_mask);
+
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr->time_to_live;
+
+ /* Perform LPM lookup to get NH and next node */
+ rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh);
+
+ /* Extract next node id and NH */
+ rte_node_mbuf_priv1(mbuf0)->nh = dst.u32[0] & 0xFFFF;
+ next0 = (dst.u32[0] >> 16);
+
+ rte_node_mbuf_priv1(mbuf1)->nh = dst.u32[1] & 0xFFFF;
+ next1 = (dst.u32[1] >> 16);
+
+ rte_node_mbuf_priv1(mbuf2)->nh = dst.u32[2] & 0xFFFF;
+ next2 = (dst.u32[2] >> 16);
+
+ rte_node_mbuf_priv1(mbuf3)->nh = dst.u32[3] & 0xFFFF;
+ next3 = (dst.u32[3] >> 16);
+
+ /* Enqueue four to next node */
+ rte_edge_t fix_spec =
+ (next_index ^ next0) | (next_index ^ next1) |
+ (next_index ^ next2) | (next_index ^ next3);
+
+ if (unlikely(fix_spec)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ /* Next0 */
+ if (next_index == next0) {
+ to_next[0] = from[0];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next0,
+ from[0]);
+ }
+
+ /* Next1 */
+ if (next_index == next1) {
+ to_next[0] = from[1];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next1,
+ from[1]);
+ }
+
+ /* Next2 */
+ if (next_index == next2) {
+ to_next[0] = from[2];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next2,
+ from[2]);
+ }
+
+ /* Next3 */
+ if (next_index == next3) {
+ to_next[0] = from[3];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next3,
+ from[3]);
+ }
+
+ from += 4;
+
+ } else {
+ last_spec += 4;
+ }
+ }
+
+ while (n_left_from > 0) {
+ uint32_t next_hop;
+
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left_from -= 1;
+
+ /* Extract DIP of mbuf0 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+ rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
+ &next_hop);
+ next_hop = (rc == 0) ? next_hop : drop_nh;
+
+ rte_node_mbuf_priv1(mbuf0)->nh = next_hop & 0xFFFF;
+ next0 = (next_hop >> 16);
+
+ if (unlikely(next_index ^ next0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ rte_node_enqueue_x1(graph, node, next0, from[0]);
+ from += 1;
+ } else {
+ last_spec += 1;
+ }
+ }
+
+ /* !!! Home run !!! */
+ if (likely(last_spec == nb_objs)) {
+ rte_node_next_stream_move(graph, node, next_index);
+ return nb_objs;
+ }
+
+ held += last_spec;
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ rte_node_next_stream_put(graph, node, next_index, held);
+
+ return nb_objs;
+}
+
#else
static uint16_t
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 21/28] node: add ipv4 rewrite node
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (19 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 20/28] node: ipv4 lookup for x86 jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 22/28] node: add ipv4 rewrite and lookup ctrl API jerinj
` (8 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Kiran Kumar K <kirankumark@marvell.com>
Add ip4 rewrite process function for ip4_rewrite
rte_node. On every packet received by this node,
header is overwritten with new data before forwarding
it to next node. Header data to overwrite with is
identified by next hop id passed in mbuf priv data
by previous node.
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
lib/librte_node/Makefile | 1 +
lib/librte_node/ip4_rewrite.c | 270 +++++++++++++++++++++++++++++
lib/librte_node/ip4_rewrite_priv.h | 55 ++++++
lib/librte_node/meson.build | 2 +-
4 files changed, 327 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_node/ip4_rewrite.c
create mode 100644 lib/librte_node/ip4_rewrite_priv.h
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index ad3f2e349..ebf473c66 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -22,6 +22,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_rx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_rewrite.c
# install header files
SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
diff --git a/lib/librte_node/ip4_rewrite.c b/lib/librte_node/ip4_rewrite.c
new file mode 100644
index 000000000..ef49ccea0
--- /dev/null
+++ b/lib/librte_node/ip4_rewrite.c
@@ -0,0 +1,270 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_ethdev.h>
+#include <rte_ether.h>
+#include <rte_graph.h>
+#include <rte_graph_worker.h>
+#include <rte_ip.h>
+#include <rte_malloc.h>
+#include <rte_mbuf.h>
+#include <rte_tcp.h>
+#include <rte_udp.h>
+#include <rte_vect.h>
+
+#include "rte_node_ip4_api.h"
+
+#include "ip4_rewrite_priv.h"
+#include "node_private.h"
+
+static struct ip4_rewrite_node_main *ip4_rewrite_nm;
+
+static uint16_t
+ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+ struct ip4_rewrite_nh_header *nh = ip4_rewrite_nm->nh;
+ uint16_t next0, next1, next2, next3, next_index;
+ struct rte_ipv4_hdr *ip0, *ip1, *ip2, *ip3;
+ uint16_t n_left_from, held = 0, last_spec = 0;
+ void *d0, *d1, *d2, *d3;
+ void **to_next, **from;
+ rte_xmm_t priv01;
+ rte_xmm_t priv23;
+ int i;
+
+ /* Speculative next as last next */
+ next_index = *(uint16_t *)node->ctx;
+ rte_prefetch0(nh);
+
+ pkts = (struct rte_mbuf **)objs;
+ from = objs;
+ n_left_from = nb_objs;
+
+ for (i = 0; i < 4 && i < n_left_from; i++)
+ rte_prefetch0(pkts[i]);
+
+ /* Get stream for the speculated next node */
+ to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+ /* Update Ethernet header of pkts */
+ while (n_left_from >= 4) {
+ if (likely(n_left_from > 7)) {
+ /* Prefetch only next-mbuf struct and priv area.
+ * Data need not be prefetched as we only write.
+ */
+ rte_prefetch0(pkts[4]);
+ rte_prefetch0(pkts[5]);
+ rte_prefetch0(pkts[6]);
+ rte_prefetch0(pkts[7]);
+ }
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+
+ pkts += 4;
+ n_left_from -= 4;
+ priv01.u64[0] = rte_node_mbuf_priv1(mbuf0)->u;
+ priv01.u64[1] = rte_node_mbuf_priv1(mbuf1)->u;
+ priv23.u64[0] = rte_node_mbuf_priv1(mbuf2)->u;
+ priv23.u64[1] = rte_node_mbuf_priv1(mbuf3)->u;
+
+ /* Increment checksum by one. */
+ priv01.u32[1] += rte_cpu_to_be_16(0x0100);
+ priv01.u32[3] += rte_cpu_to_be_16(0x0100);
+ priv23.u32[1] += rte_cpu_to_be_16(0x0100);
+ priv23.u32[3] += rte_cpu_to_be_16(0x0100);
+
+ /* Update ttl,cksum rewrite ethernet hdr on mbuf0 */
+ d0 = rte_pktmbuf_mtod(mbuf0, void *);
+ rte_memcpy(d0, nh[priv01.u16[0]].rewrite_data,
+ nh[priv01.u16[0]].rewrite_len);
+
+ next0 = nh[priv01.u16[0]].tx_node;
+ ip0 = (struct rte_ipv4_hdr *)((uint8_t *)d0 +
+ sizeof(struct rte_ether_hdr));
+ ip0->time_to_live = priv01.u16[1] - 1;
+ ip0->hdr_checksum = priv01.u16[2] + priv01.u16[3];
+
+ /* Update ttl,cksum rewrite ethernet hdr on mbuf1 */
+ d1 = rte_pktmbuf_mtod(mbuf1, void *);
+ rte_memcpy(d1, nh[priv01.u16[4]].rewrite_data,
+ nh[priv01.u16[4]].rewrite_len);
+
+ next1 = nh[priv01.u16[4]].tx_node;
+ ip1 = (struct rte_ipv4_hdr *)((uint8_t *)d1 +
+ sizeof(struct rte_ether_hdr));
+ ip1->time_to_live = priv01.u16[5] - 1;
+ ip1->hdr_checksum = priv01.u16[6] + priv01.u16[7];
+
+ /* Update ttl,cksum rewrite ethernet hdr on mbuf2 */
+ d2 = rte_pktmbuf_mtod(mbuf2, void *);
+ rte_memcpy(d2, nh[priv23.u16[0]].rewrite_data,
+ nh[priv23.u16[0]].rewrite_len);
+ next2 = nh[priv23.u16[0]].tx_node;
+ ip2 = (struct rte_ipv4_hdr *)((uint8_t *)d2 +
+ sizeof(struct rte_ether_hdr));
+ ip2->time_to_live = priv23.u16[1] - 1;
+ ip2->hdr_checksum = priv23.u16[2] + priv23.u16[3];
+
+ /* Update ttl,cksum rewrite ethernet hdr on mbuf3 */
+ d3 = rte_pktmbuf_mtod(mbuf3, void *);
+ rte_memcpy(d3, nh[priv23.u16[4]].rewrite_data,
+ nh[priv23.u16[4]].rewrite_len);
+
+ next3 = nh[priv23.u16[4]].tx_node;
+ ip3 = (struct rte_ipv4_hdr *)((uint8_t *)d3 +
+ sizeof(struct rte_ether_hdr));
+ ip3->time_to_live = priv23.u16[5] - 1;
+ ip3->hdr_checksum = priv23.u16[6] + priv23.u16[7];
+
+ /* Enqueue four to next node */
+ rte_edge_t fix_spec =
+ ((next_index == next0) && (next0 == next1) &&
+ (next1 == next2) && (next2 == next3));
+
+ if (unlikely(fix_spec == 0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ /* next0 */
+ if (next_index == next0) {
+ to_next[0] = from[0];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next0,
+ from[0]);
+ }
+
+ /* next1 */
+ if (next_index == next1) {
+ to_next[0] = from[1];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next1,
+ from[1]);
+ }
+
+ /* next2 */
+ if (next_index == next2) {
+ to_next[0] = from[2];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next2,
+ from[2]);
+ }
+
+ /* next3 */
+ if (next_index == next3) {
+ to_next[0] = from[3];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, next3,
+ from[3]);
+ }
+
+ from += 4;
+
+ /* Change speculation if last two are same */
+ if ((next_index != next3) && (next2 == next3)) {
+ /* Put the current speculated node */
+ rte_node_next_stream_put(graph, node,
+ next_index, held);
+ held = 0;
+
+ /* Get next speculated stream */
+ next_index = next3;
+ to_next = rte_node_next_stream_get(
+ graph, node, next_index, nb_objs);
+ }
+ } else {
+ last_spec += 4;
+ }
+ }
+
+ while (n_left_from > 0) {
+ uint16_t chksum;
+
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left_from -= 1;
+
+ d0 = rte_pktmbuf_mtod(mbuf0, void *);
+ rte_memcpy(d0, nh[rte_node_mbuf_priv1(mbuf0)->nh].rewrite_data,
+ nh[rte_node_mbuf_priv1(mbuf0)->nh].rewrite_len);
+
+ next0 = nh[rte_node_mbuf_priv1(mbuf0)->nh].tx_node;
+ ip0 = (struct rte_ipv4_hdr *)((uint8_t *)d0 +
+ sizeof(struct rte_ether_hdr));
+ chksum = rte_node_mbuf_priv1(mbuf0)->cksum +
+ rte_cpu_to_be_16(0x0100);
+ chksum += chksum >= 0xffff;
+ ip0->hdr_checksum = chksum;
+ ip0->time_to_live = rte_node_mbuf_priv1(mbuf0)->ttl - 1;
+
+ if (unlikely(next_index ^ next0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ rte_node_enqueue_x1(graph, node, next0, from[0]);
+ from += 1;
+ } else {
+ last_spec += 1;
+ }
+ }
+
+ /* !!! Home run !!! */
+ if (likely(last_spec == nb_objs)) {
+ rte_node_next_stream_move(graph, node, next_index);
+ return nb_objs;
+ }
+
+ held += last_spec;
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ rte_node_next_stream_put(graph, node, next_index, held);
+ /* Save the last next used */
+ *(uint16_t *)node->ctx = next_index;
+
+ return nb_objs;
+}
+
+static int
+ip4_rewrite_node_init(const struct rte_graph *graph, struct rte_node *node)
+{
+
+ RTE_SET_USED(graph);
+ RTE_SET_USED(node);
+ node_dbg("ip4_rewrite", "Initialized ip4_rewrite node initialized");
+
+ return 0;
+}
+
+static struct rte_node_register ip4_rewrite_node = {
+ .process = ip4_rewrite_node_process,
+ .name = "ip4_rewrite",
+ /* Default edge i.e '0' is pkt drop */
+ .nb_edges = 1,
+ .next_nodes = {
+ [0] = "pkt_drop",
+ },
+ .init = ip4_rewrite_node_init,
+};
+
+RTE_NODE_REGISTER(ip4_rewrite_node);
diff --git a/lib/librte_node/ip4_rewrite_priv.h b/lib/librte_node/ip4_rewrite_priv.h
new file mode 100644
index 000000000..420996a03
--- /dev/null
+++ b/lib/librte_node/ip4_rewrite_priv.h
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+#ifndef __INCLUDE_IP4_REWRITE_PRIV_H__
+#define __INCLUDE_IP4_REWRITE_PRIV_H__
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#include <rte_common.h>
+
+#define RTE_GRAPH_IP4_REWRITE_MAX_NH 64
+#define RTE_GRAPH_IP4_REWRITE_MAX_LEN 56
+
+/**
+ * @internal
+ *
+ * Ipv4 rewrite next hop header data structure. Used to store port specific
+ * rewrite data.
+ */
+struct ip4_rewrite_nh_header {
+ uint16_t rewrite_len; /**< Header rewrite length. */
+ uint16_t tx_node; /**< Tx node next index identifier. */
+ uint16_t enabled; /**< NH enable flag */
+ uint16_t rsvd;
+ union {
+ struct {
+ struct rte_ether_addr dst;
+ /**< Destination mac address. */
+ struct rte_ether_addr src;
+ /**< Source mac address. */
+ };
+ uint8_t rewrite_data[RTE_GRAPH_IP4_REWRITE_MAX_LEN];
+ /**< Generic rewrite data */
+ };
+};
+
+/**
+ * @internal
+ *
+ * Ipv4 node main data structure.
+ */
+struct ip4_rewrite_node_main {
+ struct ip4_rewrite_nh_header nh[RTE_GRAPH_IP4_REWRITE_MAX_NH];
+ /**< Array of next hop header data */
+ uint16_t next_index[RTE_MAX_ETHPORTS];
+ /**< Next index of each configured port. */
+};
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif /* __INCLUDE_IP4_REWRITE_PRIV_H__ */
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index 702d21a24..ed78791dd 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -2,7 +2,7 @@
# Copyright(C) 2020 Marvell International Ltd.
sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
- 'ethdev_ctrl.c')
+ 'ip4_rewrite.c', 'ethdev_ctrl.c')
headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
allow_experimental_apis = true
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 22/28] node: add ipv4 rewrite and lookup ctrl API
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (20 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 21/28] node: add ipv4 rewrite node jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 23/28] node: add pkt drop node jerinj
` (7 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add ip4_rewrite and ip4_lookup ctrl API. ip4_lookup ctrl
API is used to add route entries for LPM lookup with
result data containing next hop id and next proto.
ip4_rewrite ctrl API is used to add rewrite data for
every next hop.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/ethdev_ctrl.c | 18 ++++++-
lib/librte_node/ip4_lookup.c | 80 ++++++++++++++++++++++++++++
lib/librte_node/ip4_rewrite.c | 56 +++++++++++++++++++
lib/librte_node/ip4_rewrite_priv.h | 22 ++++++++
lib/librte_node/rte_node_ip4_api.h | 44 +++++++++++++++
lib/librte_node/rte_node_version.map | 2 +
6 files changed, 221 insertions(+), 1 deletion(-)
diff --git a/lib/librte_node/ethdev_ctrl.c b/lib/librte_node/ethdev_ctrl.c
index b2ac5e2c4..845d92987 100644
--- a/lib/librte_node/ethdev_ctrl.c
+++ b/lib/librte_node/ethdev_ctrl.c
@@ -11,6 +11,7 @@
#include "ethdev_rx_priv.h"
#include "ethdev_tx_priv.h"
+#include "ip4_rewrite_priv.h"
#include "node_private.h"
static struct ethdev_ctrl {
@@ -21,14 +22,17 @@ int
rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
uint16_t nb_graphs)
{
+ struct rte_node_register *ip4_rewrite_node;
struct ethdev_tx_node_main *tx_node_data;
uint16_t tx_q_used, rx_q_used, port_id;
struct rte_node_register *tx_node;
char name[RTE_NODE_NAMESIZE];
+ const char *next_nodes = name;
struct rte_mempool *mp;
+ int i, j, rc;
uint32_t id;
- int i, j;
+ ip4_rewrite_node = ip4_rewrite_node_get();
tx_node_data = ethdev_tx_node_data_get();
tx_node = ethdev_tx_node_get();
for (i = 0; i < nb_confs; i++) {
@@ -93,6 +97,18 @@ rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs,
node_dbg("ethdev", "Tx node %s-%s: is at %u", tx_node->name,
name, id);
+
+ /* Prepare the actual name of the cloned node */
+ snprintf(name, sizeof(name), "ethdev_tx-%u", port_id);
+
+ /* Add this tx port node as next to ip4_rewrite_node */
+ rte_node_edge_update(ip4_rewrite_node->id, RTE_EDGE_ID_INVALID,
+ &next_nodes, 1);
+ /* Assuming edge id is the last one alloc'ed */
+ rc = ip4_rewrite_set_next(
+ port_id, rte_node_edge_count(ip4_rewrite_node->id) - 1);
+ if (rc < 0)
+ return rc;
}
ctrl.nb_graphs = nb_graphs;
diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c
index a9e204148..0665d1ea5 100644
--- a/lib/librte_node/ip4_lookup.c
+++ b/lib/librte_node/ip4_lookup.c
@@ -28,6 +28,8 @@ struct ip4_lookup_node_main {
struct rte_lpm *lpm_tbl[RTE_MAX_NUMA_NODES];
};
+static struct ip4_lookup_node_main ip4_lookup_nm;
+
#if defined(RTE_MACHINE_CPUFLAG_NEON)
/* ARM64 NEON */
static uint16_t
@@ -512,12 +514,90 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
#endif
+int
+rte_node_ip4_route_add(uint32_t ip, uint8_t depth, uint16_t next_hop,
+ enum rte_node_ip4_lookup_next next_node)
+{
+ char abuf[INET6_ADDRSTRLEN];
+ struct in_addr in;
+ uint8_t socket;
+ uint32_t val;
+ int ret;
+
+ in.s_addr = htonl(ip);
+ inet_ntop(AF_INET, &in, abuf, sizeof(abuf));
+ /* Embedded next node id in next hop */
+ val = (next_node << 16) | next_hop;
+ node_dbg("ip4_lookup", "LPM: Adding route %s / %d nh (0x%x)", abuf,
+ depth, val);
+
+ for (socket = 0; socket < RTE_MAX_NUMA_NODES; socket++) {
+ if (!ip4_lookup_nm.lpm_tbl[socket])
+ continue;
+
+ ret = rte_lpm_add(ip4_lookup_nm.lpm_tbl[socket], ip, depth,
+ val);
+
+ if (ret < 0) {
+ node_err("ip4_lookup",
+ "Unable to add entry %s / %d nh (%x) to LPM table on sock %d, rc=%d\n",
+ abuf, depth, val, socket, ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int
+setup_lpm(struct ip4_lookup_node_main *nm, int socket)
+{
+ struct rte_lpm_config config_ipv4;
+ char s[RTE_LPM_NAMESIZE];
+
+ /* One LPM table per socket */
+ if (nm->lpm_tbl[socket])
+ return 0;
+
+ /* create the LPM table */
+ config_ipv4.max_rules = IPV4_L3FWD_LPM_MAX_RULES;
+ config_ipv4.number_tbl8s = IPV4_L3FWD_LPM_NUMBER_TBL8S;
+ config_ipv4.flags = 0;
+ snprintf(s, sizeof(s), "IPV4_L3FWD_LPM_%d", socket);
+ nm->lpm_tbl[socket] = rte_lpm_create(s, socket, &config_ipv4);
+ if (nm->lpm_tbl[socket] == NULL)
+ return -rte_errno;
+
+ return 0;
+}
+
static int
ip4_lookup_node_init(const struct rte_graph *graph, struct rte_node *node)
{
+ struct rte_lpm **lpm_p = (struct rte_lpm **)&node->ctx;
+ uint16_t socket, lcore_id;
+ static uint8_t init_once;
+ int rc;
+
RTE_SET_USED(graph);
RTE_SET_USED(node);
+ if (!init_once) {
+ /* Setup LPM tables for all sockets */
+ RTE_LCORE_FOREACH(lcore_id)
+ {
+ socket = rte_lcore_to_socket_id(lcore_id);
+ rc = setup_lpm(&ip4_lookup_nm, socket);
+ if (rc) {
+ node_err("ip4_lookup",
+ "Failed to setup lpm tbl for sock %u, rc=%d",
+ socket, rc);
+ return rc;
+ }
+ }
+ init_once = 1;
+ }
+ *lpm_p = ip4_lookup_nm.lpm_tbl[graph->socket];
node_dbg("ip4_lookup", "Initialized ip4_lookup node");
return 0;
diff --git a/lib/librte_node/ip4_rewrite.c b/lib/librte_node/ip4_rewrite.c
index ef49ccea0..5663f1eb1 100644
--- a/lib/librte_node/ip4_rewrite.c
+++ b/lib/librte_node/ip4_rewrite.c
@@ -256,6 +256,56 @@ ip4_rewrite_node_init(const struct rte_graph *graph, struct rte_node *node)
return 0;
}
+int
+ip4_rewrite_set_next(uint16_t port_id, uint16_t next_index)
+{
+ if (ip4_rewrite_nm == NULL) {
+ ip4_rewrite_nm = rte_zmalloc(
+ "ip4_rewrite", sizeof(struct ip4_rewrite_node_main),
+ RTE_CACHE_LINE_SIZE);
+ if (ip4_rewrite_nm == NULL)
+ return -ENOMEM;
+ }
+ ip4_rewrite_nm->next_index[port_id] = next_index;
+
+ return 0;
+}
+
+int
+rte_node_ip4_rewrite_add(uint16_t next_hop, uint8_t *rewrite_data,
+ uint8_t rewrite_len, uint16_t dst_port)
+{
+ struct ip4_rewrite_nh_header *nh;
+
+ if (next_hop >= RTE_GRAPH_IP4_REWRITE_MAX_NH)
+ return -EINVAL;
+
+ if (rewrite_len > RTE_GRAPH_IP4_REWRITE_MAX_LEN)
+ return -EINVAL;
+
+ if (ip4_rewrite_nm == NULL) {
+ ip4_rewrite_nm = rte_zmalloc(
+ "ip4_rewrite", sizeof(struct ip4_rewrite_node_main),
+ RTE_CACHE_LINE_SIZE);
+ if (ip4_rewrite_nm == NULL)
+ return -ENOMEM;
+ }
+
+ /* Check if dst port doesn't exist as edge */
+ if (!ip4_rewrite_nm->next_index[dst_port])
+ return -EINVAL;
+
+ /* Update next hop */
+ nh = &ip4_rewrite_nm->nh[next_hop];
+
+ memcpy(nh->rewrite_data, rewrite_data, rewrite_len);
+ nh->tx_node = ip4_rewrite_nm->next_index[dst_port];
+ nh->rewrite_len = rewrite_len;
+ nh->enabled = true;
+
+ return 0;
+}
+
static struct rte_node_register ip4_rewrite_node = {
.process = ip4_rewrite_node_process,
.name = "ip4_rewrite",
@@ -267,4 +317,10 @@ static struct rte_node_register ip4_rewrite_node = {
.init = ip4_rewrite_node_init,
};
+struct rte_node_register *
+ip4_rewrite_node_get(void)
+{
+ return &ip4_rewrite_node;
+}
+
RTE_NODE_REGISTER(ip4_rewrite_node);
diff --git a/lib/librte_node/ip4_rewrite_priv.h b/lib/librte_node/ip4_rewrite_priv.h
index 420996a03..80f0abdc9 100644
--- a/lib/librte_node/ip4_rewrite_priv.h
+++ b/lib/librte_node/ip4_rewrite_priv.h
@@ -48,6 +48,28 @@ struct ip4_rewrite_node_main {
/**< Next index of each configured port. */
};
+/**
+ * @internal
+ *
+ * Get the ipv4 rewrite node.
+ *
+ * @retrun
+ * Pointer to the ipv4 rewrite node.
+ */
+struct rte_node_register *ip4_rewrite_node_get(void);
+
+/**
+ * @internal
+ *
+ * Set the Edge index of a given port_id.
+ *
+ * @param port_id
+ * Ethernet port identifier.
+ * @param next_index
+ * Edge index of the Given Tx node.
+ */
+int ip4_rewrite_set_next(uint16_t port_id, uint16_t next_index);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/librte_node/rte_node_ip4_api.h b/lib/librte_node/rte_node_ip4_api.h
index 37c12bf82..394cac097 100644
--- a/lib/librte_node/rte_node_ip4_api.h
+++ b/lib/librte_node/rte_node_ip4_api.h
@@ -36,6 +36,50 @@ enum rte_node_ip4_lookup_next {
/**< Number of next nodes of lookup node. */
};
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add ipv4 route to lookup table.
+ *
+ * @param ip
+ * IP address of route to be added.
+ * @param depth
+ * Depth of the rule to be added.
+ * @param next_hop
+ * Next hop id of the rule result to be added.
+ * @param next_node
+ * Next node to redirect traffic to.
+ *
+ * @return
+ * 0 on success, negative otherwise.
+ */
+__rte_experimental
+int rte_node_ip4_route_add(uint32_t ip, uint8_t depth, uint16_t next_hop,
+ enum rte_node_ip4_lookup_next next_node);
+
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice
+ *
+ * Add a next hop's rewrite data.
+ *
+ * @param next_hop
+ * Next hop id to add rewrite data to.
+ * @param rewrite_data
+ * Rewrite data.
+ * @param rewrite_len
+ * Length of rewrite data.
+ * @param dst_port
+ * Destination port to redirect traffic to.
+ *
+ * @return
+ * 0 on success, negative otherwise.
+ */
+__rte_experimental
+int rte_node_ip4_rewrite_add(uint16_t next_hop, uint8_t *rewrite_data,
+ uint8_t rewrite_len, uint16_t dst_port);
+
#ifdef __cplusplus
}
#endif
diff --git a/lib/librte_node/rte_node_version.map b/lib/librte_node/rte_node_version.map
index c6c71bd02..a799b0d38 100644
--- a/lib/librte_node/rte_node_version.map
+++ b/lib/librte_node/rte_node_version.map
@@ -2,6 +2,8 @@ EXPERIMENTAL {
global:
rte_node_eth_config;
+ rte_node_ip4_route_add;
+ rte_node_ip4_rewrite_add;
rte_node_logtype;
local: *;
};
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 23/28] node: add pkt drop node
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (21 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 22/28] node: add ipv4 rewrite and lookup ctrl API jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 24/28] l3fwd-graph: add graph based l3fwd skeleton jerinj
` (6 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Nithin Dabilpuram, Pavan Nikhilesh
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add pkt drop node process function for pkt_drop
rte_node. This node simply free's every object received as
an rte_mbuf to its rte_pktmbuf pool.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
---
lib/librte_node/Makefile | 1 +
lib/librte_node/meson.build | 2 +-
lib/librte_node/pkt_drop.c | 26 ++++++++++++++++++++++++++
3 files changed, 28 insertions(+), 1 deletion(-)
create mode 100644 lib/librte_node/pkt_drop.c
diff --git a/lib/librte_node/Makefile b/lib/librte_node/Makefile
index ebf473c66..322b651c8 100644
--- a/lib/librte_node/Makefile
+++ b/lib/librte_node/Makefile
@@ -23,6 +23,7 @@ SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_tx.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ethdev_ctrl.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_lookup.c
SRCS-$(CONFIG_RTE_LIBRTE_NODE) += ip4_rewrite.c
+SRCS-$(CONFIG_RTE_LIBRTE_NODE) += pkt_drop.c
# install header files
SYMLINK-$(CONFIG_RTE_LIBRTE_NODE)-include += rte_node_ip4_api.h
diff --git a/lib/librte_node/meson.build b/lib/librte_node/meson.build
index ed78791dd..8aa93654b 100644
--- a/lib/librte_node/meson.build
+++ b/lib/librte_node/meson.build
@@ -2,7 +2,7 @@
# Copyright(C) 2020 Marvell International Ltd.
sources = files('null.c', 'log.c', 'ethdev_rx.c', 'ethdev_tx.c', 'ip4_lookup.c',
- 'ip4_rewrite.c', 'ethdev_ctrl.c')
+ 'ip4_rewrite.c', 'pkt_drop.c', 'ethdev_ctrl.c')
headers = files('rte_node_ip4_api.h', 'rte_node_eth_api.h')
allow_experimental_apis = true
# Strict-aliasing rules are violated by uint8_t[] to context size casts.
diff --git a/lib/librte_node/pkt_drop.c b/lib/librte_node/pkt_drop.c
new file mode 100644
index 000000000..c35001323
--- /dev/null
+++ b/lib/librte_node/pkt_drop.c
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <rte_debug.h>
+#include <rte_graph.h>
+#include <rte_mbuf.h>
+
+static uint16_t
+pkt_drop_process(struct rte_graph *graph, struct rte_node *node, void **objs,
+ uint16_t nb_objs)
+{
+ RTE_SET_USED(node);
+ RTE_SET_USED(graph);
+
+ rte_pktmbuf_free_bulk((struct rte_mbuf **)objs, nb_objs);
+
+ return nb_objs;
+}
+
+static struct rte_node_register pkt_drop_node = {
+ .process = pkt_drop_process,
+ .name = "pkt_drop",
+};
+
+RTE_NODE_REGISTER(pkt_drop_node);
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 24/28] l3fwd-graph: add graph based l3fwd skeleton
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (22 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 23/28] node: add pkt drop node jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 25/28] l3fwd-graph: add ethdev configuration changes jerinj
` (5 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Thomas Monjalon, Marko Kovacevic, Ori Kam, Bruce Richardson,
Radu Nicolau, Akhil Goyal, Tomasz Kantecki, Sunil Kumar Kori,
Pavan Nikhilesh, Nithin Dabilpuram
Cc: dev, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add graph based l3fwd application skeleton with cmdline
parsing support inline with normal l3fwd.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
MAINTAINERS | 3 +
examples/Makefile | 3 +
examples/l3fwd-graph/Makefile | 58 ++++
examples/l3fwd-graph/main.c | 525 +++++++++++++++++++++++++++++++
examples/l3fwd-graph/meson.build | 13 +
examples/meson.build | 6 +-
6 files changed, 606 insertions(+), 2 deletions(-)
create mode 100644 examples/l3fwd-graph/Makefile
create mode 100644 examples/l3fwd-graph/main.c
create mode 100644 examples/l3fwd-graph/meson.build
diff --git a/MAINTAINERS b/MAINTAINERS
index c1acaedab..1d2cf6caa 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1581,6 +1581,9 @@ F: doc/guides/sample_app_ug/l2_forward_event.rst
F: examples/l3fwd/
F: doc/guides/sample_app_ug/l3_forward.rst
+M: Nithin Dabilpuram <ndabilpuram@marvell.com>
+F: examples/l3fwd-graph/
+
F: examples/link_status_interrupt/
F: doc/guides/sample_app_ug/link_status_intr.rst
diff --git a/examples/Makefile b/examples/Makefile
index feff79784..b7e99a2f7 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -51,6 +51,9 @@ DIRS-$(CONFIG_RTE_LIBRTE_ACL) += l3fwd-acl
ifeq ($(CONFIG_RTE_LIBRTE_LPM)$(CONFIG_RTE_LIBRTE_HASH),yy)
DIRS-$(CONFIG_RTE_LIBRTE_POWER) += l3fwd-power
endif
+ifeq ($(CONFIG_RTE_LIBRTE_GRAPH),y)
+DIRS-y += l3fwd-graph
+endif
DIRS-y += link_status_interrupt
DIRS-y += multi_process
DIRS-y += ntb
diff --git a/examples/l3fwd-graph/Makefile b/examples/l3fwd-graph/Makefile
new file mode 100644
index 000000000..f3cf275ec
--- /dev/null
+++ b/examples/l3fwd-graph/Makefile
@@ -0,0 +1,58 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+# binary name
+APP = l3fwd-graph
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+# Build using pkg-config variables if possible
+ifeq ($(shell pkg-config --exists libdpdk && echo 0),0)
+
+all: shared
+.PHONY: shared static
+shared: build/$(APP)-shared
+ ln -sf $(APP)-shared build/$(APP)
+static: build/$(APP)-static
+ ln -sf $(APP)-static build/$(APP)
+
+PKGCONF=pkg-config --define-prefix
+
+PC_FILE := $(shell $(PKGCONF) --path libdpdk)
+CFLAGS += -O3 $(shell $(PKGCONF) --cflags libdpdk) -DALLOW_EXPERIMENTAL_API
+LDFLAGS_SHARED = $(shell $(PKGCONF) --libs libdpdk)
+LDFLAGS_STATIC = -Wl,-Bstatic $(shell $(PKGCONF) --static --libs libdpdk)
+
+build/$(APP)-shared: $(SRCS-y) Makefile $(PC_FILE) | build
+ $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_SHARED)
+
+build/$(APP)-static: $(SRCS-y) Makefile $(PC_FILE) | build
+ $(CC) $(CFLAGS) $(SRCS-y) -o $@ $(LDFLAGS) $(LDFLAGS_STATIC)
+
+build:
+ @mkdir -p $@
+
+.PHONY: clean
+clean:
+ rm -f build/$(APP) build/$(APP)-static build/$(APP)-shared
+ test -d build && rmdir -p build || true
+
+else # Build using legacy build system
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, detect a build directory, by looking for a path with a .config
+RTE_TARGET ?= $(notdir $(abspath $(dir $(firstword $(wildcard $(RTE_SDK)/*/.config)))))
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+CFLAGS += -I$(SRCDIR)
+CFLAGS += -O3 $(USER_FLAGS)
+CFLAGS += $(WERROR_FLAGS)
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+endif
diff --git a/examples/l3fwd-graph/main.c b/examples/l3fwd-graph/main.c
new file mode 100644
index 000000000..e0c6f42fa
--- /dev/null
+++ b/examples/l3fwd-graph/main.c
@@ -0,0 +1,525 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#include <arpa/inet.h>
+#include <errno.h>
+#include <getopt.h>
+#include <inttypes.h>
+#include <signal.h>
+#include <stdarg.h>
+#include <stdbool.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/socket.h>
+#include <sys/types.h>
+#include <sys/queue.h>
+#include <unistd.h>
+
+#include <rte_branch_prediction.h>
+#include <rte_common.h>
+#include <rte_eal.h>
+#include <rte_ethdev.h>
+#include <rte_log.h>
+#include <rte_mempool.h>
+#include <rte_per_lcore.h>
+#include <rte_string_fns.h>
+#include <rte_vect.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_parse_etheraddr.h>
+
+/* Log type */
+#define RTE_LOGTYPE_L3FWD_GRAPH RTE_LOGTYPE_USER1
+
+/*
+ * Configurable number of RX/TX ring descriptors
+ */
+#define RTE_TEST_RX_DESC_DEFAULT 1024
+#define RTE_TEST_TX_DESC_DEFAULT 1024
+
+#define MAX_TX_QUEUE_PER_PORT RTE_MAX_ETHPORTS
+#define MAX_RX_QUEUE_PER_PORT 128
+
+#define MAX_RX_QUEUE_PER_LCORE 16
+
+#define MAX_LCORE_PARAMS 1024
+
+#define NB_SOCKETS 8
+
+/**< Ports set in promiscuous mode off by default. */
+static int promiscuous_on;
+
+static int numa_on = 1; /**< NUMA is enabled by default. */
+static int per_port_pool; /**< Use separate buffer pools per port; disabled */
+ /**< by default */
+
+static volatile bool force_quit;
+
+/* Ethernet addresses of ports */
+static uint64_t dest_eth_addr[RTE_MAX_ETHPORTS];
+xmm_t val_eth[RTE_MAX_ETHPORTS];
+
+/* Mask of enabled ports */
+static uint32_t enabled_port_mask;
+
+struct lcore_rx_queue {
+ uint16_t port_id;
+ uint8_t queue_id;
+};
+
+/* Lcore conf */
+struct lcore_conf {
+ uint16_t n_rx_queue;
+ struct lcore_rx_queue rx_queue_list[MAX_RX_QUEUE_PER_LCORE];
+} __rte_cache_aligned;
+
+static struct lcore_conf lcore_conf[RTE_MAX_LCORE];
+
+struct lcore_params {
+ uint16_t port_id;
+ uint8_t queue_id;
+ uint8_t lcore_id;
+} __rte_cache_aligned;
+
+static struct lcore_params lcore_params_array[MAX_LCORE_PARAMS];
+static struct lcore_params lcore_params_array_default[] = {
+ {0, 0, 2}, {0, 1, 2}, {0, 2, 2}, {1, 0, 2}, {1, 1, 2},
+ {1, 2, 2}, {2, 0, 2}, {3, 0, 3}, {3, 1, 3},
+};
+
+static struct lcore_params *lcore_params = lcore_params_array_default;
+static uint16_t nb_lcore_params = RTE_DIM(lcore_params_array_default);
+
+static struct rte_eth_conf port_conf = {
+ .rxmode = {
+ .mq_mode = ETH_MQ_RX_RSS,
+ .max_rx_pkt_len = RTE_ETHER_MAX_LEN,
+ .split_hdr_size = 0,
+ },
+ .rx_adv_conf = {
+ .rss_conf = {
+ .rss_key = NULL,
+ .rss_hf = ETH_RSS_IP,
+ },
+ },
+ .txmode = {
+ .mq_mode = ETH_MQ_TX_NONE,
+ },
+};
+
+static int
+check_lcore_params(void)
+{
+ uint8_t queue, lcore;
+ int socketid;
+ uint16_t i;
+
+ for (i = 0; i < nb_lcore_params; ++i) {
+ queue = lcore_params[i].queue_id;
+ if (queue >= MAX_RX_QUEUE_PER_PORT) {
+ printf("Invalid queue number: %hhu\n", queue);
+ return -1;
+ }
+ lcore = lcore_params[i].lcore_id;
+ if (!rte_lcore_is_enabled(lcore)) {
+ printf("Error: lcore %hhu is not enabled in lcore mask\n",
+ lcore);
+ return -1;
+ }
+
+ if (lcore == rte_get_master_lcore()) {
+ printf("Error: lcore %u is master lcore\n", lcore);
+ return -1;
+ }
+ socketid = rte_lcore_to_socket_id(lcore);
+ if ((socketid != 0) && (numa_on == 0)) {
+ printf("Warning: lcore %hhu is on socket %d with numa off\n",
+ lcore, socketid);
+ }
+ }
+
+ return 0;
+}
+
+static int
+check_port_config(void)
+{
+ uint16_t portid;
+ uint16_t i;
+
+ for (i = 0; i < nb_lcore_params; ++i) {
+ portid = lcore_params[i].port_id;
+ if ((enabled_port_mask & (1 << portid)) == 0) {
+ printf("Port %u is not enabled in port mask\n", portid);
+ return -1;
+ }
+ if (!rte_eth_dev_is_valid_port(portid)) {
+ printf("Port %u is not present on the board\n", portid);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static int
+init_lcore_rx_queues(void)
+{
+ uint16_t i, nb_rx_queue;
+ uint8_t lcore;
+
+ for (i = 0; i < nb_lcore_params; ++i) {
+ lcore = lcore_params[i].lcore_id;
+ nb_rx_queue = lcore_conf[lcore].n_rx_queue;
+ if (nb_rx_queue >= MAX_RX_QUEUE_PER_LCORE) {
+ printf("Error: too many queues (%u) for lcore: %u\n",
+ (unsigned int)nb_rx_queue + 1,
+ (unsigned int)lcore);
+ return -1;
+ }
+
+ lcore_conf[lcore].rx_queue_list[nb_rx_queue].port_id =
+ lcore_params[i].port_id;
+ lcore_conf[lcore].rx_queue_list[nb_rx_queue].queue_id =
+ lcore_params[i].queue_id;
+ lcore_conf[lcore].n_rx_queue++;
+ }
+
+ return 0;
+}
+
+/* Display usage */
+static void
+print_usage(const char *prgname)
+{
+ fprintf(stderr,
+ "%s [EAL options] --"
+ " -p PORTMASK"
+ " [-P]"
+ " --config (port,queue,lcore)[,(port,queue,lcore)]"
+ " [--eth-dest=X,MM:MM:MM:MM:MM:MM]"
+ " [--enable-jumbo [--max-pkt-len PKTLEN]]"
+ " [--no-numa]"
+ " [--per-port-pool]\n\n"
+
+ " -p PORTMASK: Hexadecimal bitmask of ports to configure\n"
+ " -P : Enable promiscuous mode\n"
+ " --config (port,queue,lcore): Rx queue configuration\n"
+ " --eth-dest=X,MM:MM:MM:MM:MM:MM: Ethernet destination for "
+ "port X\n"
+ " --enable-jumbo: Enable jumbo frames\n"
+ " --max-pkt-len: Under the premise of enabling jumbo,\n"
+ " maximum packet length in decimal (64-9600)\n"
+ " --no-numa: Disable numa awareness\n"
+ " --per-port-pool: Use separate buffer pool per port\n\n",
+ prgname);
+}
+
+static int
+parse_max_pkt_len(const char *pktlen)
+{
+ unsigned long len;
+ char *end = NULL;
+
+ /* Parse decimal string */
+ len = strtoul(pktlen, &end, 10);
+ if ((pktlen[0] == '\0') || (end == NULL) || (*end != '\0'))
+ return -1;
+
+ if (len == 0)
+ return -1;
+
+ return len;
+}
+
+static int
+parse_portmask(const char *portmask)
+{
+ char *end = NULL;
+ unsigned long pm;
+
+ /* Parse hexadecimal string */
+ pm = strtoul(portmask, &end, 16);
+ if ((portmask[0] == '\0') || (end == NULL) || (*end != '\0'))
+ return -1;
+
+ if (pm == 0)
+ return -1;
+
+ return pm;
+}
+
+static int
+parse_config(const char *q_arg)
+{
+ enum fieldnames { FLD_PORT = 0, FLD_QUEUE, FLD_LCORE, _NUM_FLD };
+ unsigned long int_fld[_NUM_FLD];
+ const char *p, *p0 = q_arg;
+ char *str_fld[_NUM_FLD];
+ uint32_t size;
+ char s[256];
+ char *end;
+ int i;
+
+ nb_lcore_params = 0;
+
+ while ((p = strchr(p0, '(')) != NULL) {
+ ++p;
+ p0 = strchr(p, ')');
+ if (p0 == NULL)
+ return -1;
+
+ size = p0 - p;
+ if (size >= sizeof(s))
+ return -1;
+
+ snprintf(s, sizeof(s), "%.*s", size, p);
+ if (rte_strsplit(s, sizeof(s), str_fld, _NUM_FLD, ',') !=
+ _NUM_FLD)
+ return -1;
+ for (i = 0; i < _NUM_FLD; i++) {
+ errno = 0;
+ int_fld[i] = strtoul(str_fld[i], &end, 0);
+ if (errno != 0 || end == str_fld[i] || int_fld[i] > 255)
+ return -1;
+ }
+ if (nb_lcore_params >= MAX_LCORE_PARAMS) {
+ printf("Exceeded max number of lcore params: %hu\n",
+ nb_lcore_params);
+ return -1;
+ }
+ lcore_params_array[nb_lcore_params].port_id =
+ (uint8_t)int_fld[FLD_PORT];
+ lcore_params_array[nb_lcore_params].queue_id =
+ (uint8_t)int_fld[FLD_QUEUE];
+ lcore_params_array[nb_lcore_params].lcore_id =
+ (uint8_t)int_fld[FLD_LCORE];
+ ++nb_lcore_params;
+ }
+ lcore_params = lcore_params_array;
+
+ return 0;
+}
+
+static void
+parse_eth_dest(const char *optarg)
+{
+ uint8_t c, *dest, peer_addr[6];
+ uint16_t portid;
+ char *port_end;
+
+ errno = 0;
+ portid = strtoul(optarg, &port_end, 10);
+ if (errno != 0 || port_end == optarg || *port_end++ != ',')
+ rte_exit(EXIT_FAILURE, "Invalid eth-dest: %s", optarg);
+ if (portid >= RTE_MAX_ETHPORTS)
+ rte_exit(EXIT_FAILURE,
+ "eth-dest: port %d >= RTE_MAX_ETHPORTS(%d)\n", portid,
+ RTE_MAX_ETHPORTS);
+
+ if (cmdline_parse_etheraddr(NULL, port_end, &peer_addr,
+ sizeof(peer_addr)) < 0)
+ rte_exit(EXIT_FAILURE, "Invalid ethernet address: %s\n",
+ port_end);
+ dest = (uint8_t *)&dest_eth_addr[portid];
+ for (c = 0; c < 6; c++)
+ dest[c] = peer_addr[c];
+ *(uint64_t *)(val_eth + portid) = dest_eth_addr[portid];
+}
+
+#define MAX_JUMBO_PKT_LEN 9600
+#define MEMPOOL_CACHE_SIZE 256
+
+static const char short_options[] = "p:" /* portmask */
+ "P" /* promiscuous */
+ ;
+
+#define CMD_LINE_OPT_CONFIG "config"
+#define CMD_LINE_OPT_ETH_DEST "eth-dest"
+#define CMD_LINE_OPT_NO_NUMA "no-numa"
+#define CMD_LINE_OPT_ENABLE_JUMBO "enable-jumbo"
+#define CMD_LINE_OPT_PER_PORT_POOL "per-port-pool"
+enum {
+ /* Long options mapped to a short option */
+
+ /* First long only option value must be >= 256, so that we won't
+ * conflict with short options
+ */
+ CMD_LINE_OPT_MIN_NUM = 256,
+ CMD_LINE_OPT_CONFIG_NUM,
+ CMD_LINE_OPT_ETH_DEST_NUM,
+ CMD_LINE_OPT_NO_NUMA_NUM,
+ CMD_LINE_OPT_ENABLE_JUMBO_NUM,
+ CMD_LINE_OPT_PARSE_PER_PORT_POOL,
+};
+
+static const struct option lgopts[] = {
+ {CMD_LINE_OPT_CONFIG, 1, 0, CMD_LINE_OPT_CONFIG_NUM},
+ {CMD_LINE_OPT_ETH_DEST, 1, 0, CMD_LINE_OPT_ETH_DEST_NUM},
+ {CMD_LINE_OPT_NO_NUMA, 0, 0, CMD_LINE_OPT_NO_NUMA_NUM},
+ {CMD_LINE_OPT_ENABLE_JUMBO, 0, 0, CMD_LINE_OPT_ENABLE_JUMBO_NUM},
+ {CMD_LINE_OPT_PER_PORT_POOL, 0, 0, CMD_LINE_OPT_PARSE_PER_PORT_POOL},
+ {NULL, 0, 0, 0},
+};
+
+/*
+ * This expression is used to calculate the number of mbufs needed
+ * depending on user input, taking into account memory for rx and
+ * tx hardware rings, cache per lcore and mtable per port per lcore.
+ * RTE_MAX is used to ensure that NB_MBUF never goes below a minimum
+ * value of 8192
+ */
+#define NB_MBUF(nports) \
+ RTE_MAX((nports * nb_rx_queue * nb_rxd + \
+ nports * nb_lcores * RTE_GRAPH_BURST_SIZE + \
+ nports * n_tx_queue * nb_txd + \
+ nb_lcores * MEMPOOL_CACHE_SIZE), 8192u)
+
+/* Parse the argument given in the command line of the application */
+static int
+parse_args(int argc, char **argv)
+{
+ char *prgname = argv[0];
+ int option_index;
+ char **argvopt;
+ int opt, ret;
+
+ argvopt = argv;
+
+ /* Error or normal output strings. */
+ while ((opt = getopt_long(argc, argvopt, short_options, lgopts,
+ &option_index)) != EOF) {
+
+ switch (opt) {
+ /* Portmask */
+ case 'p':
+ enabled_port_mask = parse_portmask(optarg);
+ if (enabled_port_mask == 0) {
+ fprintf(stderr, "Invalid portmask\n");
+ print_usage(prgname);
+ return -1;
+ }
+ break;
+
+ case 'P':
+ promiscuous_on = 1;
+ break;
+
+ /* Long options */
+ case CMD_LINE_OPT_CONFIG_NUM:
+ ret = parse_config(optarg);
+ if (ret) {
+ fprintf(stderr, "Invalid config\n");
+ print_usage(prgname);
+ return -1;
+ }
+ break;
+
+ case CMD_LINE_OPT_ETH_DEST_NUM:
+ parse_eth_dest(optarg);
+ break;
+
+ case CMD_LINE_OPT_NO_NUMA_NUM:
+ numa_on = 0;
+ break;
+
+ case CMD_LINE_OPT_ENABLE_JUMBO_NUM: {
+ const struct option lenopts = {"max-pkt-len",
+ required_argument, 0, 0};
+
+ port_conf.rxmode.offloads |= DEV_RX_OFFLOAD_JUMBO_FRAME;
+ port_conf.txmode.offloads |= DEV_TX_OFFLOAD_MULTI_SEGS;
+
+ /*
+ * if no max-pkt-len set, use the default
+ * value RTE_ETHER_MAX_LEN.
+ */
+ if (getopt_long(argc, argvopt, "", &lenopts,
+ &option_index) == 0) {
+ ret = parse_max_pkt_len(optarg);
+ if (ret < 64 || ret > MAX_JUMBO_PKT_LEN) {
+ fprintf(stderr, "Invalid maximum "
+ "packet length\n");
+ print_usage(prgname);
+ return -1;
+ }
+ port_conf.rxmode.max_rx_pkt_len = ret;
+ }
+ break;
+ }
+
+ case CMD_LINE_OPT_PARSE_PER_PORT_POOL:
+ printf("Per port buffer pool is enabled\n");
+ per_port_pool = 1;
+ break;
+
+ default:
+ print_usage(prgname);
+ return -1;
+ }
+ }
+
+ if (optind >= 0)
+ argv[optind - 1] = prgname;
+ ret = optind - 1;
+ optind = 1; /* Reset getopt lib */
+
+ return ret;
+}
+
+static void
+signal_handler(int signum)
+{
+ if (signum == SIGINT || signum == SIGTERM) {
+ printf("\n\nSignal %d received, preparing to exit...\n",
+ signum);
+ force_quit = true;
+ }
+}
+
+int
+main(int argc, char **argv)
+{
+ uint16_t portid;
+ int ret;
+
+ /* Init EAL */
+ ret = rte_eal_init(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Invalid EAL parameters\n");
+ argc -= ret;
+ argv += ret;
+
+ force_quit = false;
+ signal(SIGINT, signal_handler);
+ signal(SIGTERM, signal_handler);
+
+ /* Pre-init dst MACs for all ports to 02:00:00:00:00:xx */
+ for (portid = 0; portid < RTE_MAX_ETHPORTS; portid++) {
+ dest_eth_addr[portid] =
+ RTE_ETHER_LOCAL_ADMIN_ADDR + ((uint64_t)portid << 40);
+ *(uint64_t *)(val_eth + portid) = dest_eth_addr[portid];
+ }
+
+ /* Parse application arguments (after the EAL ones) */
+ ret = parse_args(argc, argv);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "Invalid L3FWD_GRAPH parameters\n");
+
+ if (check_lcore_params() < 0)
+ rte_exit(EXIT_FAILURE, "check_lcore_params() failed\n");
+
+ ret = init_lcore_rx_queues();
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "init_lcore_rx_queues() failed\n");
+
+ if (check_port_config() < 0)
+ rte_exit(EXIT_FAILURE, "check_port_config() failed\n");
+
+ printf("Bye...\n");
+
+ return ret;
+}
diff --git a/examples/l3fwd-graph/meson.build b/examples/l3fwd-graph/meson.build
new file mode 100644
index 000000000..a816bd890
--- /dev/null
+++ b/examples/l3fwd-graph/meson.build
@@ -0,0 +1,13 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(C) 2020 Marvell International Ltd.
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+deps += ['graph', 'eal', 'lpm', 'ethdev', 'node' ]
+sources = files(
+ 'main.c'
+)
+allow_experimental_apis = true
diff --git a/examples/meson.build b/examples/meson.build
index 1f2b6f516..3b540012f 100644
--- a/examples/meson.build
+++ b/examples/meson.build
@@ -2,8 +2,10 @@
# Copyright(c) 2017-2019 Intel Corporation
driver_libs = []
+node_libs = []
if get_option('default_library') == 'static'
driver_libs = dpdk_drivers
+ node_libs = dpdk_graph_nodes
endif
execinfo = cc.find_library('execinfo', required: false)
@@ -23,7 +25,7 @@ all_examples = [
'l2fwd', 'l2fwd-cat', 'l2fwd-event',
'l2fwd-crypto', 'l2fwd-jobstats',
'l2fwd-keepalive', 'l3fwd',
- 'l3fwd-acl', 'l3fwd-power',
+ 'l3fwd-acl', 'l3fwd-power', 'l3fwd-graph',
'link_status_interrupt',
'multi_process/client_server_mp/mp_client',
'multi_process/client_server_mp/mp_server',
@@ -99,7 +101,7 @@ foreach example: examples
endif
executable('dpdk-' + name, sources,
include_directories: includes,
- link_whole: driver_libs,
+ link_whole: driver_libs + node_libs,
link_args: dpdk_extra_ldflags,
c_args: cflags,
dependencies: dep_objs)
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 25/28] l3fwd-graph: add ethdev configuration changes
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (23 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 24/28] l3fwd-graph: add graph based l3fwd skeleton jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 26/28] l3fwd-graph: add graph config and main loop jerinj
` (4 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Marko Kovacevic, Ori Kam, Bruce Richardson, Radu Nicolau,
Akhil Goyal, Tomasz Kantecki, Sunil Kumar Kori, Pavan Nikhilesh,
Nithin Dabilpuram
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add changes to ethdev port and queue configuration based
on command line parameters for l3fwd graph application.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
examples/l3fwd-graph/main.c | 350 +++++++++++++++++++++++++++++++++++-
1 file changed, 349 insertions(+), 1 deletion(-)
diff --git a/examples/l3fwd-graph/main.c b/examples/l3fwd-graph/main.c
index e0c6f42fa..47d2f2ecb 100644
--- a/examples/l3fwd-graph/main.c
+++ b/examples/l3fwd-graph/main.c
@@ -20,8 +20,10 @@
#include <rte_branch_prediction.h>
#include <rte_common.h>
+#include <rte_cycles.h>
#include <rte_eal.h>
#include <rte_ethdev.h>
+#include <rte_lcore.h>
#include <rte_log.h>
#include <rte_mempool.h>
#include <rte_per_lcore.h>
@@ -49,6 +51,10 @@
#define NB_SOCKETS 8
+/* Static global variables used within this file. */
+static uint16_t nb_rxd = RTE_TEST_RX_DESC_DEFAULT;
+static uint16_t nb_txd = RTE_TEST_TX_DESC_DEFAULT;
+
/**< Ports set in promiscuous mode off by default. */
static int promiscuous_on;
@@ -60,6 +66,7 @@ static volatile bool force_quit;
/* Ethernet addresses of ports */
static uint64_t dest_eth_addr[RTE_MAX_ETHPORTS];
+static struct rte_ether_addr ports_eth_addr[RTE_MAX_ETHPORTS];
xmm_t val_eth[RTE_MAX_ETHPORTS];
/* Mask of enabled ports */
@@ -110,6 +117,8 @@ static struct rte_eth_conf port_conf = {
},
};
+static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
+
static int
check_lcore_params(void)
{
@@ -165,6 +174,27 @@ check_port_config(void)
return 0;
}
+static uint8_t
+get_port_n_rx_queues(const uint16_t port)
+{
+ int queue = -1;
+ uint16_t i;
+
+ for (i = 0; i < nb_lcore_params; ++i) {
+ if (lcore_params[i].port_id == port) {
+ if (lcore_params[i].queue_id == queue + 1)
+ queue = lcore_params[i].queue_id;
+ else
+ rte_exit(EXIT_FAILURE,
+ "Queue ids of the port %d must be"
+ " in sequence and must start with 0\n",
+ lcore_params[i].port_id);
+ }
+ }
+
+ return (uint8_t)(++queue);
+}
+
static int
init_lcore_rx_queues(void)
{
@@ -470,6 +500,120 @@ parse_args(int argc, char **argv)
return ret;
}
+static void
+print_ethaddr(const char *name, const struct rte_ether_addr *eth_addr)
+{
+ char buf[RTE_ETHER_ADDR_FMT_SIZE];
+ rte_ether_format_addr(buf, RTE_ETHER_ADDR_FMT_SIZE, eth_addr);
+ printf("%s%s", name, buf);
+}
+
+static int
+init_mem(uint16_t portid, uint32_t nb_mbuf)
+{
+ uint32_t lcore_id;
+ int socketid;
+ char s[64];
+
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+
+ if (numa_on)
+ socketid = rte_lcore_to_socket_id(lcore_id);
+ else
+ socketid = 0;
+
+ if (socketid >= NB_SOCKETS) {
+ rte_exit(EXIT_FAILURE,
+ "Socket %d of lcore %u is out of range %d\n",
+ socketid, lcore_id, NB_SOCKETS);
+ }
+
+ if (pktmbuf_pool[portid][socketid] == NULL) {
+ snprintf(s, sizeof(s), "mbuf_pool_%d:%d", portid,
+ socketid);
+ /* Create a pool with priv size of a cacheline */
+ pktmbuf_pool[portid][socketid] =
+ rte_pktmbuf_pool_create(
+ s, nb_mbuf, MEMPOOL_CACHE_SIZE,
+ RTE_CACHE_LINE_SIZE,
+ RTE_MBUF_DEFAULT_BUF_SIZE, socketid);
+ if (pktmbuf_pool[portid][socketid] == NULL)
+ rte_exit(EXIT_FAILURE,
+ "Cannot init mbuf pool on socket %d\n",
+ socketid);
+ else
+ printf("Allocated mbuf pool on socket %d\n",
+ socketid);
+ }
+ }
+
+ return 0;
+}
+
+/* Check the link status of all ports in up to 9s, and print them finally */
+static void
+check_all_ports_link_status(uint32_t port_mask)
+{
+#define CHECK_INTERVAL 100 /* 100ms */
+#define MAX_CHECK_TIME 90 /* 9s (90 * 100ms) in total */
+ uint8_t count, all_ports_up, print_flag = 0;
+ struct rte_eth_link link;
+ uint16_t portid;
+
+ printf("\nChecking link status");
+ fflush(stdout);
+ for (count = 0; count <= MAX_CHECK_TIME; count++) {
+ if (force_quit)
+ return;
+ all_ports_up = 1;
+ RTE_ETH_FOREACH_DEV(portid)
+ {
+ if (force_quit)
+ return;
+ if ((port_mask & (1 << portid)) == 0)
+ continue;
+ memset(&link, 0, sizeof(link));
+ rte_eth_link_get_nowait(portid, &link);
+ /* Print link status if flag set */
+ if (print_flag == 1) {
+ if (link.link_status)
+ printf("Port%d Link Up. Speed %u Mbps "
+ "-%s\n",
+ portid, link.link_speed,
+ (link.link_duplex ==
+ ETH_LINK_FULL_DUPLEX)
+ ? ("full-duplex")
+ : ("half-duplex\n"));
+ else
+ printf("Port %d Link Down\n", portid);
+ continue;
+ }
+ /* Clear all_ports_up flag if any link down */
+ if (link.link_status == ETH_LINK_DOWN) {
+ all_ports_up = 0;
+ break;
+ }
+ }
+ /* After finally printing all link status, get out */
+ if (print_flag == 1)
+ break;
+
+ if (all_ports_up == 0) {
+ printf(".");
+ fflush(stdout);
+ rte_delay_ms(CHECK_INTERVAL);
+ }
+
+ /* Set the print_flag if all ports up or timeout */
+ if (all_ports_up == 1 || count == (MAX_CHECK_TIME - 1)) {
+ print_flag = 1;
+ printf("Done\n");
+ }
+ }
+}
+
static void
signal_handler(int signum)
{
@@ -483,7 +627,14 @@ signal_handler(int signum)
int
main(int argc, char **argv)
{
- uint16_t portid;
+ uint8_t nb_rx_queue, queue, socketid;
+ struct rte_eth_dev_info dev_info;
+ uint32_t n_tx_queue, nb_lcores;
+ struct rte_eth_txconf *txconf;
+ uint16_t queueid, portid;
+ struct lcore_conf *qconf;
+ uint32_t lcore_id;
+ uint32_t nb_ports;
int ret;
/* Init EAL */
@@ -519,6 +670,203 @@ main(int argc, char **argv)
if (check_port_config() < 0)
rte_exit(EXIT_FAILURE, "check_port_config() failed\n");
+ nb_ports = rte_eth_dev_count_avail();
+ nb_lcores = rte_lcore_count();
+
+ /* Initialize all ports */
+ RTE_ETH_FOREACH_DEV(portid)
+ {
+ struct rte_eth_conf local_port_conf = port_conf;
+
+ /* Skip ports that are not enabled */
+ if ((enabled_port_mask & (1 << portid)) == 0) {
+ printf("\nSkipping disabled port %d\n", portid);
+ continue;
+ }
+
+ /* Init port */
+ printf("Initializing port %d ... ", portid);
+ fflush(stdout);
+
+ nb_rx_queue = get_port_n_rx_queues(portid);
+ n_tx_queue = nb_lcores;
+ if (n_tx_queue > MAX_TX_QUEUE_PER_PORT)
+ n_tx_queue = MAX_TX_QUEUE_PER_PORT;
+ printf("Creating queues: nb_rxq=%d nb_txq=%u... ",
+ nb_rx_queue, n_tx_queue);
+
+ rte_eth_dev_info_get(portid, &dev_info);
+ if (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_MBUF_FAST_FREE)
+ local_port_conf.txmode.offloads |=
+ DEV_TX_OFFLOAD_MBUF_FAST_FREE;
+
+ local_port_conf.rx_adv_conf.rss_conf.rss_hf &=
+ dev_info.flow_type_rss_offloads;
+ if (local_port_conf.rx_adv_conf.rss_conf.rss_hf !=
+ port_conf.rx_adv_conf.rss_conf.rss_hf) {
+ printf("Port %u modified RSS hash function based on "
+ "hardware support,"
+ "requested:%#" PRIx64 " configured:%#" PRIx64
+ "\n",
+ portid, port_conf.rx_adv_conf.rss_conf.rss_hf,
+ local_port_conf.rx_adv_conf.rss_conf.rss_hf);
+ }
+
+ ret = rte_eth_dev_configure(portid, nb_rx_queue,
+ n_tx_queue, &local_port_conf);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "Cannot configure device: err=%d, port=%d\n",
+ ret, portid);
+
+ ret = rte_eth_dev_adjust_nb_rx_tx_desc(portid, &nb_rxd,
+ &nb_txd);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "Cannot adjust number of descriptors: err=%d, "
+ "port=%d\n",
+ ret, portid);
+
+ rte_eth_macaddr_get(portid, &ports_eth_addr[portid]);
+ print_ethaddr(" Address:", &ports_eth_addr[portid]);
+ printf(", ");
+ print_ethaddr(
+ "Destination:",
+ (const struct rte_ether_addr *)&dest_eth_addr[portid]);
+ printf(", ");
+
+ /*
+ * prepare src MACs for each port.
+ */
+ rte_ether_addr_copy(
+ &ports_eth_addr[portid],
+ (struct rte_ether_addr *)(val_eth + portid) + 1);
+
+ /* Init memory */
+ if (!per_port_pool) {
+ /* portid = 0; this is *not* signifying the first port,
+ * rather, it signifies that portid is ignored.
+ */
+ ret = init_mem(0, NB_MBUF(nb_ports));
+ } else {
+ ret = init_mem(portid, NB_MBUF(1));
+ }
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE, "init_mem() failed\n");
+
+ /* Init one TX queue per couple (lcore,port) */
+ queueid = 0;
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+
+ qconf = &lcore_conf[lcore_id];
+
+ if (numa_on)
+ socketid = (uint8_t)rte_lcore_to_socket_id(
+ lcore_id);
+ else
+ socketid = 0;
+
+ printf("txq=%u,%d,%d ", lcore_id, queueid, socketid);
+ fflush(stdout);
+
+ txconf = &dev_info.default_txconf;
+ txconf->offloads = local_port_conf.txmode.offloads;
+ ret = rte_eth_tx_queue_setup(portid, queueid, nb_txd,
+ socketid, txconf);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "rte_eth_tx_queue_setup: err=%d, "
+ "port=%d\n",
+ ret, portid);
+ queueid++;
+ }
+
+ printf("\n");
+ }
+
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+ qconf = &lcore_conf[lcore_id];
+ printf("\nInitializing rx queues on lcore %u ... ", lcore_id);
+ fflush(stdout);
+ /* Init RX queues */
+ for (queue = 0; queue < qconf->n_rx_queue; ++queue) {
+ struct rte_eth_rxconf rxq_conf;
+
+ portid = qconf->rx_queue_list[queue].port_id;
+ queueid = qconf->rx_queue_list[queue].queue_id;
+
+ if (numa_on)
+ socketid = (uint8_t)rte_lcore_to_socket_id(
+ lcore_id);
+ else
+ socketid = 0;
+
+ printf("rxq=%d,%d,%d ", portid, queueid, socketid);
+ fflush(stdout);
+
+ rte_eth_dev_info_get(portid, &dev_info);
+ rxq_conf = dev_info.default_rxconf;
+ rxq_conf.offloads = port_conf.rxmode.offloads;
+ if (!per_port_pool)
+ ret = rte_eth_rx_queue_setup(
+ portid, queueid, nb_rxd, socketid,
+ &rxq_conf, pktmbuf_pool[0][socketid]);
+ else
+ ret = rte_eth_rx_queue_setup(
+ portid, queueid, nb_rxd, socketid,
+ &rxq_conf,
+ pktmbuf_pool[portid][socketid]);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "rte_eth_rx_queue_setup: err=%d, "
+ "port=%d\n",
+ ret, portid);
+
+ }
+ }
+
+ printf("\n");
+
+ /* Start ports */
+ RTE_ETH_FOREACH_DEV(portid)
+ {
+ if ((enabled_port_mask & (1 << portid)) == 0)
+ continue;
+
+ /* Start device */
+ ret = rte_eth_dev_start(portid);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "rte_eth_dev_start: err=%d, port=%d\n", ret,
+ portid);
+
+ /*
+ * If enabled, put device in promiscuous mode.
+ * This allows IO forwarding mode to forward packets
+ * to itself through 2 cross-connected ports of the
+ * target machine.
+ */
+ if (promiscuous_on)
+ rte_eth_promiscuous_enable(portid);
+ }
+
+ printf("\n");
+
+ check_all_ports_link_status(enabled_port_mask);
+
+ /* Stop ports */
+ RTE_ETH_FOREACH_DEV(portid) {
+ if ((enabled_port_mask & (1 << portid)) == 0)
+ continue;
+ printf("Closing port %d...", portid);
+ rte_eth_dev_stop(portid);
+ rte_eth_dev_close(portid);
+ printf(" Done\n");
+ }
printf("Bye...\n");
return ret;
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 26/28] l3fwd-graph: add graph config and main loop
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (24 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 25/28] l3fwd-graph: add ethdev configuration changes jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 27/28] doc: add graph library programmer's guide guide jerinj
` (3 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: Marko Kovacevic, Ori Kam, Bruce Richardson, Radu Nicolau,
Akhil Goyal, Tomasz Kantecki, Sunil Kumar Kori, Pavan Nikhilesh,
Nithin Dabilpuram
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add graph creation, configuration logic and graph main loop.
This graph main loop is run on every slave lcore and calls
rte_graph_walk() to walk over lcore specific rte_graph.
Master core accumulates and prints graph walk stats of all the
lcore's graph's.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
---
examples/l3fwd-graph/main.c | 242 +++++++++++++++++++++++++++++++++++-
1 file changed, 240 insertions(+), 2 deletions(-)
diff --git a/examples/l3fwd-graph/main.c b/examples/l3fwd-graph/main.c
index 47d2f2ecb..28db947b5 100644
--- a/examples/l3fwd-graph/main.c
+++ b/examples/l3fwd-graph/main.c
@@ -23,9 +23,13 @@
#include <rte_cycles.h>
#include <rte_eal.h>
#include <rte_ethdev.h>
+#include <rte_graph_worker.h>
+#include <rte_launch.h>
#include <rte_lcore.h>
#include <rte_log.h>
#include <rte_mempool.h>
+#include <rte_node_eth_api.h>
+#include <rte_node_ip4_api.h>
#include <rte_per_lcore.h>
#include <rte_string_fns.h>
#include <rte_vect.h>
@@ -75,12 +79,17 @@ static uint32_t enabled_port_mask;
struct lcore_rx_queue {
uint16_t port_id;
uint8_t queue_id;
+ char node_name[RTE_NODE_NAMESIZE];
};
/* Lcore conf */
struct lcore_conf {
uint16_t n_rx_queue;
struct lcore_rx_queue rx_queue_list[MAX_RX_QUEUE_PER_LCORE];
+
+ struct rte_graph *graph;
+ char name[RTE_GRAPH_NAMESIZE];
+ rte_graph_t graph_id;
} __rte_cache_aligned;
static struct lcore_conf lcore_conf[RTE_MAX_LCORE];
@@ -119,6 +128,25 @@ static struct rte_eth_conf port_conf = {
static struct rte_mempool *pktmbuf_pool[RTE_MAX_ETHPORTS][NB_SOCKETS];
+static struct rte_node_ethdev_config ethdev_conf[RTE_MAX_ETHPORTS];
+
+struct ipv4_l3fwd_lpm_route {
+ uint32_t ip;
+ uint8_t depth;
+ uint8_t if_out;
+};
+
+#define IPV4_L3FWD_LPM_NUM_ROUTES \
+ (sizeof(ipv4_l3fwd_lpm_route_array) / \
+ sizeof(ipv4_l3fwd_lpm_route_array[0]))
+/* 198.18.0.0/16 are set aside for RFC2544 benchmarking. */
+static struct ipv4_l3fwd_lpm_route ipv4_l3fwd_lpm_route_array[] = {
+ {RTE_IPV4(198, 18, 0, 0), 24, 0}, {RTE_IPV4(198, 18, 1, 0), 24, 1},
+ {RTE_IPV4(198, 18, 2, 0), 24, 2}, {RTE_IPV4(198, 18, 3, 0), 24, 3},
+ {RTE_IPV4(198, 18, 4, 0), 24, 4}, {RTE_IPV4(198, 18, 5, 0), 24, 5},
+ {RTE_IPV4(198, 18, 6, 0), 24, 6}, {RTE_IPV4(198, 18, 7, 0), 24, 7},
+};
+
static int
check_lcore_params(void)
{
@@ -624,17 +652,87 @@ signal_handler(int signum)
}
}
+static void
+print_stats(void)
+{
+ const char topLeft[] = {27, '[', '1', ';', '1', 'H', '\0'};
+ const char clr[] = {27, '[', '2', 'J', '\0'};
+ struct rte_graph_cluster_stats_param s_param;
+ struct rte_graph_cluster_stats *stats;
+ const char *pattern = "worker_*";
+
+ /* Prepare stats object */
+ memset(&s_param, 0, sizeof(s_param));
+ s_param.f = stdout;
+ s_param.socket_id = SOCKET_ID_ANY;
+ s_param.graph_patterns = &pattern;
+ s_param.nb_graph_patterns = 1;
+
+ stats = rte_graph_cluster_stats_create(&s_param);
+ if (stats == NULL)
+ rte_exit(EXIT_FAILURE, "Unable to create stats object\n");
+
+ while (!force_quit) {
+ /* Clear screen and move to top left */
+ printf("%s%s", clr, topLeft);
+ rte_graph_cluster_stats_get(stats, 0);
+ rte_delay_ms(1E3);
+ }
+
+ rte_graph_cluster_stats_destroy(stats);
+}
+
+/* Main processing loop */
+static int
+graph_main_loop(void *conf)
+{
+ struct lcore_conf *qconf;
+ struct rte_graph *graph;
+ uint32_t lcore_id;
+
+ RTE_SET_USED(conf);
+
+ lcore_id = rte_lcore_id();
+ qconf = &lcore_conf[lcore_id];
+ graph = qconf->graph;
+
+ if (!graph) {
+ RTE_LOG(INFO, L3FWD_GRAPH, "Lcore %u has nothing to do\n",
+ lcore_id);
+ return 0;
+ }
+
+ RTE_LOG(INFO, L3FWD_GRAPH,
+ "Entering main loop on lcore %u, graph %s(%p)\n", lcore_id,
+ qconf->name, graph);
+
+ while (likely(!force_quit))
+ rte_graph_walk(graph);
+
+ return 0;
+}
+
int
main(int argc, char **argv)
{
+ uint8_t rewrite_data[2 * sizeof(struct rte_ether_addr)];
+ static const char *node_patterns[] = {
+ "ip4*",
+ "ethdev_tx-*",
+ "pkt_drop",
+ };
uint8_t nb_rx_queue, queue, socketid;
+ struct rte_graph_param graph_conf;
struct rte_eth_dev_info dev_info;
+ uint32_t nb_ports, nb_conf = 0;
uint32_t n_tx_queue, nb_lcores;
struct rte_eth_txconf *txconf;
- uint16_t queueid, portid;
+ uint16_t queueid, portid, i;
struct lcore_conf *qconf;
+ uint16_t nb_graphs = 0;
+ uint16_t nb_patterns;
+ uint8_t rewrite_len;
uint32_t lcore_id;
- uint32_t nb_ports;
int ret;
/* Init EAL */
@@ -783,6 +881,18 @@ main(int argc, char **argv)
queueid++;
}
+ /* Setup ethdev node config */
+ ethdev_conf[nb_conf].port_id = portid;
+ ethdev_conf[nb_conf].num_rx_queues = nb_rx_queue;
+ ethdev_conf[nb_conf].num_tx_queues = n_tx_queue;
+ if (!per_port_pool)
+ ethdev_conf[nb_conf].mp = pktmbuf_pool[0];
+
+ else
+ ethdev_conf[nb_conf].mp = pktmbuf_pool[portid];
+ ethdev_conf[nb_conf].mp_count = NB_SOCKETS;
+
+ nb_conf++;
printf("\n");
}
@@ -826,11 +936,26 @@ main(int argc, char **argv)
"port=%d\n",
ret, portid);
+ /* Add this queue node to its graph */
+ snprintf(qconf->rx_queue_list[queue].node_name,
+ RTE_NODE_NAMESIZE, "ethdev_rx-%u-%u", portid,
+ queueid);
+ }
+
+ /* Alloc a graph to this lcore only if source exists */
+ if (qconf->n_rx_queue) {
+ qconf->graph_id = nb_graphs;
+ nb_graphs++;
}
}
printf("\n");
+ /* Ethdev node config, skip rx queue mapping */
+ ret = rte_node_eth_config(ethdev_conf, nb_conf, nb_graphs);
+ if (ret)
+ rte_exit(EXIT_FAILURE, "rte_node_eth_config: err=%d\n", ret);
+
/* Start ports */
RTE_ETH_FOREACH_DEV(portid)
{
@@ -858,6 +983,119 @@ main(int argc, char **argv)
check_all_ports_link_status(enabled_port_mask);
+ /* Graph Initialization */
+ memset(&graph_conf, 0, sizeof(graph_conf));
+ graph_conf.node_patterns = node_patterns;
+ nb_patterns = RTE_DIM(node_patterns);
+
+ for (lcore_id = 0; lcore_id < RTE_MAX_LCORE; lcore_id++) {
+ rte_graph_t graph_id;
+ rte_edge_t i;
+
+ if (rte_lcore_is_enabled(lcore_id) == 0)
+ continue;
+
+ qconf = &lcore_conf[lcore_id];
+
+ /* Skip graph creation if no source exists */
+ if (!qconf->n_rx_queue)
+ continue;
+
+ /* Add rx node patterns of this lcore */
+ for (i = 0; i < qconf->n_rx_queue; i++) {
+ graph_conf.node_patterns[nb_patterns + i] =
+ qconf->rx_queue_list[i].node_name;
+ }
+
+ graph_conf.nb_node_patterns = nb_patterns + i;
+ graph_conf.socket_id = rte_lcore_to_socket_id(lcore_id);
+
+ snprintf(qconf->name, sizeof(qconf->name), "worker_%u",
+ lcore_id);
+
+ graph_id = rte_graph_create(qconf->name, &graph_conf);
+ if (graph_id != qconf->graph_id)
+ rte_exit(EXIT_FAILURE,
+ "rte_graph_create(): graph_id=%d not "
+ " as expected for lcore %u(%u\n",
+ graph_id, lcore_id, qconf->graph_id);
+
+ qconf->graph = rte_graph_lookup(qconf->name);
+ if (!qconf->graph)
+ rte_exit(EXIT_FAILURE,
+ "rte_graph_lookup(): graph %s not found\n",
+ qconf->name);
+ }
+
+ memset(&rewrite_data, 0, sizeof(rewrite_data));
+ rewrite_len = sizeof(rewrite_data);
+
+ /* Add route to ip4 graph infra */
+ for (i = 0; i < IPV4_L3FWD_LPM_NUM_ROUTES; i++) {
+ char route_str[INET6_ADDRSTRLEN * 4];
+ char abuf[INET6_ADDRSTRLEN];
+ struct in_addr in;
+ uint32_t dst_port;
+ uint16_t next_hop;
+
+ /* Skip unused ports */
+ if ((1 << ipv4_l3fwd_lpm_route_array[i].if_out &
+ enabled_port_mask) == 0)
+ continue;
+
+ dst_port = ipv4_l3fwd_lpm_route_array[i].if_out;
+ next_hop = i;
+
+ in.s_addr = htonl(ipv4_l3fwd_lpm_route_array[i].ip);
+ snprintf(route_str, sizeof(route_str), "%s / %d (%d)",
+ inet_ntop(AF_INET, &in, abuf, sizeof(abuf)),
+ ipv4_l3fwd_lpm_route_array[i].depth,
+ ipv4_l3fwd_lpm_route_array[i].if_out);
+
+ ret = rte_node_ip4_route_add(
+ ipv4_l3fwd_lpm_route_array[i].ip,
+ ipv4_l3fwd_lpm_route_array[i].depth, next_hop,
+ RTE_NODE_IP4_LOOKUP_NEXT_REWRITE);
+
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "Unable to add ip4 route %s to graph\n",
+ route_str);
+
+ memcpy(rewrite_data, val_eth + dst_port, rewrite_len);
+
+ /* Add next hop for a given destination */
+ ret = rte_node_ip4_rewrite_add(next_hop, rewrite_data,
+ rewrite_len, dst_port);
+ if (ret < 0)
+ rte_exit(EXIT_FAILURE,
+ "Unable to add next hop %u for "
+ "route %s\n",
+ next_hop, route_str);
+
+ RTE_LOG(INFO, L3FWD_GRAPH, "Added route %s, next_hop %u\n",
+ route_str, next_hop);
+ }
+
+ /* Launch per-lcore init on every slave lcore */
+ rte_eal_mp_remote_launch(graph_main_loop, NULL, SKIP_MASTER);
+
+ /* Accumulate and print stats on master until exit */
+ if (rte_graph_has_stats_feature())
+ print_stats();
+
+ /* Wait for slave cores to exit */
+ ret = 0;
+ RTE_LCORE_FOREACH_SLAVE(lcore_id) {
+ ret = rte_eal_wait_lcore(lcore_id);
+ /* Destroy graph */
+ rte_graph_destroy(lcore_conf[lcore_id].name);
+ if (ret < 0) {
+ ret = -1;
+ break;
+ }
+ }
+
/* Stop ports */
RTE_ETH_FOREACH_DEV(portid) {
if ((enabled_port_mask & (1 << portid)) == 0)
--
2.25.1
^ permalink raw reply [flat|nested] 219+ messages in thread
* [dpdk-dev] [PATCH v2 27/28] doc: add graph library programmer's guide guide
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 00/28] graph: introduce graph subsystem jerinj
` (25 preceding siblings ...)
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 26/28] l3fwd-graph: add graph config and main loop jerinj
@ 2020-03-26 16:56 ` jerinj
2020-03-26 16:56 ` [dpdk-dev] [PATCH v2 28/28] doc: add l3fwd graph application user guide jerinj
` (2 subsequent siblings)
29 siblings, 0 replies; 219+ messages in thread
From: jerinj @ 2020-03-26 16:56 UTC (permalink / raw)
To: John McNamara, Marko Kovacevic
Cc: dev, thomas, david.marchand, mdr, mattias.ronnblom, kirankumark,
pbhagavatula, ndabilpuram, Jerin Jacob
From: Jerin Jacob <jerinj@marvell.com>
Adding programmer's guide for Graph library and the inbuilt nodes.
This patch also updates the release note for the new libraries.
Signed-off-by: Jerin Jacob <jerinj@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---