From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 0AE7C45FD0; Fri, 3 Jan 2025 07:06:52 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 205A74060F; Fri, 3 Jan 2025 07:06:32 +0100 (CET) Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com [67.231.148.174]) by mails.dpdk.org (Postfix) with ESMTP id 86656402DE for ; Fri, 3 Jan 2025 07:06:30 +0100 (CET) Received: from pps.filterd (m0045849.ppops.net [127.0.0.1]) by mx0a-0016f401.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 5035tL3P020773; Thu, 2 Jan 2025 22:06:27 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s=pfpt0220; bh=T bTl+e4q3upQHvdxfsiXwSFqGWMltkAMActZFadNOu8=; b=NyXCoeWgleKVCnSbk ETLW1OfgVId8VaYrTCaZ0doGZphf2TGnI8cJmRj33Cs+pD/O4/UDP0+k0lqxAjEx zyeCfDLxHxSsRoXo+TX1w1/eMkMQzecFatnBXIxVSKFXv1fkV/7lExRzFsS1uNN6 uw0wqW62c/tPIBLo64ebWWNCVk9jwUD4OEPvk54LAW9kBErC9mLYqsrfuSk+xH0u QK+/lvS2WCltLZ1hqqFd6/+uC+WpZPGzB3yns2T06bcpQmSV6cjo37DJWf5UDTPA Lg7I4gT6SNjy34veEL82c09eGusXmN3BzrGHRoupVdXB2DxDyVrVJNMN71xCGlXM +t0bw== Received: from dc6wp-exch02.marvell.com ([4.21.29.225]) by mx0a-0016f401.pphosted.com (PPS) with ESMTPS id 43xa5s00gj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 02 Jan 2025 22:06:27 -0800 (PST) Received: from DC6WP-EXCH02.marvell.com (10.76.176.209) by DC6WP-EXCH02.marvell.com (10.76.176.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Thu, 2 Jan 2025 22:06:26 -0800 Received: from maili.marvell.com (10.69.176.80) by DC6WP-EXCH02.marvell.com (10.76.176.209) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Thu, 2 Jan 2025 22:06:26 -0800 Received: from cavium-PowerEdge-R640.. (unknown [10.28.36.207]) by maili.marvell.com (Postfix) with ESMTP id 73E3B3F7088; Thu, 2 Jan 2025 22:06:23 -0800 (PST) From: Nitin Saxena To: Jerin Jacob , Kiran Kumar K , Nithin Dabilpuram , Zhirun Yan , Robin Jarry , Christophe Fontaine CC: , Nitin Saxena Subject: [PATCH v6 3/4] ip4: add ip4 output feature arc Date: Fri, 3 Jan 2025 11:36:06 +0530 Message-ID: <20250103060612.2671836-4-nsaxena@marvell.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250103060612.2671836-1-nsaxena@marvell.com> References: <20241014143401.3135897-1-nsaxena@marvell.com> <20250103060612.2671836-1-nsaxena@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-GUID: GOSE59ug1IJNydV4LEsYGhJWy-MRoujx X-Proofpoint-ORIG-GUID: GOSE59ug1IJNydV4LEsYGhJWy-MRoujx X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.60.29 definitions=2024-09-06_09,2024-09-06_01,2024-09-02_01 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Added ip4 output arc to allow applications to hook feature nodes in ip4 egress direction Signed-off-by: Nitin Saxena --- lib/node/ethdev_ctrl.c | 8 + lib/node/interface_tx_feature.c | 133 ++++++++++++ lib/node/interface_tx_feature_priv.h | 33 +++ lib/node/ip4_rewrite.c | 298 ++++++++++++++++++++++++++- lib/node/meson.build | 1 + lib/node/node_private.h | 1 + lib/node/rte_node_ip4_api.h | 4 + 7 files changed, 474 insertions(+), 4 deletions(-) create mode 100644 lib/node/interface_tx_feature.c create mode 100644 lib/node/interface_tx_feature_priv.h diff --git a/lib/node/ethdev_ctrl.c b/lib/node/ethdev_ctrl.c index cd52e8be08..93ef7fbb95 100644 --- a/lib/node/ethdev_ctrl.c +++ b/lib/node/ethdev_ctrl.c @@ -14,6 +14,7 @@ #include "ethdev_tx_priv.h" #include "ip4_rewrite_priv.h" #include "ip6_rewrite_priv.h" +#include "interface_tx_feature_priv.h" #include "node_private.h" static struct ethdev_ctrl { @@ -24,6 +25,7 @@ int rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs, uint16_t nb_graphs) { + struct rte_node_register *if_tx_feature_node; struct rte_node_register *ip4_rewrite_node; struct rte_node_register *ip6_rewrite_node; struct ethdev_tx_node_main *tx_node_data; @@ -35,6 +37,7 @@ rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs, int i, j, rc; uint32_t id; + if_tx_feature_node = if_tx_feature_node_get(); ip4_rewrite_node = ip4_rewrite_node_get(); ip6_rewrite_node = ip6_rewrite_node_get(); tx_node_data = ethdev_tx_node_data_get(); @@ -125,6 +128,11 @@ rte_node_eth_config(struct rte_node_ethdev_config *conf, uint16_t nb_confs, if (rc < 0) return rc; + /* Add this tx port node to if_tx_feature_node */ + rte_node_edge_update(if_tx_feature_node->id, RTE_EDGE_ID_INVALID, + &next_nodes, 1); + rc = if_tx_feature_node_set_next(port_id, + rte_node_edge_count(if_tx_feature_node->id) - 1); } ctrl.nb_graphs = nb_graphs; diff --git a/lib/node/interface_tx_feature.c b/lib/node/interface_tx_feature.c new file mode 100644 index 0000000000..35ac00f21e --- /dev/null +++ b/lib/node/interface_tx_feature.c @@ -0,0 +1,133 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2025 Marvell International Ltd. + */ + +#include +#include + +#include +#include +#include +#include + +#include "rte_node_ip4_api.h" +#include "node_private.h" +#include "interface_tx_feature_priv.h" + +#define IF_TX_FEATURE_LAST_NEXT_INDEX(ctx) \ + (((struct if_tx_feature_node_ctx *)ctx)->last_index) +/* + * @internal array for mapping port to next node index + */ +struct if_tx_feature_node_main { + uint16_t next_index[RTE_MAX_ETHPORTS]; +}; + +struct if_tx_feature_node_ctx { + uint16_t last_index; +}; + +static struct if_tx_feature_node_main *if_tx_feature_nm; + +int +if_tx_feature_node_set_next(uint16_t port_id, uint16_t next_index) +{ + if (if_tx_feature_nm == NULL) { + if_tx_feature_nm = rte_zmalloc( + "if_tx_feature_nm", sizeof(struct if_tx_feature_node_main), + RTE_CACHE_LINE_SIZE); + if (if_tx_feature_nm == NULL) + return -ENOMEM; + } + if_tx_feature_nm->next_index[port_id] = next_index; + + return 0; +} + +static int +if_tx_feature_node_init(const struct rte_graph *graph, struct rte_node *node) +{ + RTE_SET_USED(graph); + + /* pkt_drop */ + IF_TX_FEATURE_LAST_NEXT_INDEX(node->ctx) = 0; + + return 0; +} + +static uint16_t +if_tx_feature_node_process(struct rte_graph *graph, struct rte_node *node, + void **objs, uint16_t nb_objs) +{ + uint16_t held = 0, next; + void **to_next, **from; + uint16_t last_spec = 0; + rte_edge_t next_index; + struct rte_mbuf *mbuf; + int i; + + /* Speculative next */ + next_index = IF_TX_FEATURE_LAST_NEXT_INDEX(node->ctx); + + from = objs; + to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs); + for (i = 0; i < nb_objs; i++) { + + mbuf = (struct rte_mbuf *)objs[i]; + + /* port-tx node starts from next edge 1*/ + next = if_tx_feature_nm->next_index[mbuf->port]; + + if (unlikely(next_index != next)) { + /* Copy things successfully speculated till now */ + rte_memcpy(to_next, from, last_spec * sizeof(from[0])); + from += last_spec; + to_next += last_spec; + held += last_spec; + last_spec = 0; + + rte_node_enqueue_x1(graph, node, next, from[0]); + from += 1; + } else { + last_spec += 1; + } + } + /* !!! Home run !!! */ + if (likely(last_spec == nb_objs)) { + rte_node_next_stream_move(graph, node, next_index); + return nb_objs; + } + held += last_spec; + rte_memcpy(to_next, from, last_spec * sizeof(from[0])); + rte_node_next_stream_put(graph, node, next_index, held); + + IF_TX_FEATURE_LAST_NEXT_INDEX(node->ctx) = next; + + return nb_objs; +} + +static struct rte_node_register if_tx_feature_node = { + .process = if_tx_feature_node_process, + .init = if_tx_feature_node_init, + .name = "interface_tx", + .nb_edges = 1, + .next_nodes = { + [0] = "pkt_drop", + }, +}; + +struct rte_node_register * +if_tx_feature_node_get(void) +{ + return &if_tx_feature_node; +} + +RTE_NODE_REGISTER(if_tx_feature_node); + +/* if_tx feature node */ +struct rte_graph_feature_register if_tx_feature = { + .feature_name = RTE_IP4_OUTPUT_END_FEATURE_NAME, + .arc_name = RTE_IP4_OUTPUT_FEATURE_ARC_NAME, + .feature_process_fn = if_tx_feature_node_process, + .feature_node = &if_tx_feature_node, +}; diff --git a/lib/node/interface_tx_feature_priv.h b/lib/node/interface_tx_feature_priv.h new file mode 100644 index 0000000000..846191572d --- /dev/null +++ b/lib/node/interface_tx_feature_priv.h @@ -0,0 +1,33 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(C) 2025 Marvell International Ltd. + */ +#ifndef __INCLUDE_IF_TX_FEATURE_PRIV_H__ +#define __INCLUDE_IF_TX_FEATURE_PRIV_H__ + +#include + +extern struct rte_graph_feature_register if_tx_feature; + +/** + * @internal + * + * Get the ipv4 rewrite node. + * + * @return + * Pointer to the ipv4 rewrite node. + */ +struct rte_node_register *if_tx_feature_node_get(void); + +/** + * @internal + * + * Set the Edge index of a given port_id. + * + * @param port_id + * Ethernet port identifier. + * @param next_index + * Edge index of the Given Tx node. + */ +int if_tx_feature_node_set_next(uint16_t port_id, uint16_t next_index); + +#endif /* __INCLUDE_INTERFCE_TX_FEATURE_PRIV_H */ diff --git a/lib/node/ip4_rewrite.c b/lib/node/ip4_rewrite.c index 34a920df5e..ab64aa0a3c 100644 --- a/lib/node/ip4_rewrite.c +++ b/lib/node/ip4_rewrite.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include #include @@ -14,15 +15,27 @@ #include "ip4_rewrite_priv.h" #include "node_private.h" +#include "interface_tx_feature_priv.h" + +#ifndef RTE_IP4_OUTPUT_ARC_INDEXES +#define RTE_IP4_OUTPUT_ARC_INDEXES RTE_MAX_ETHPORTS +#endif struct ip4_rewrite_node_ctx { /* Dynamic offset to mbuf priv1 */ int mbuf_priv1_off; + /* Dynamic offset to feature arc field */ + int arc_dyn_off; /* Cached next index */ uint16_t next_index; + /* tx interface of last mbuf */ + uint16_t last_tx_if; + /* Cached feature arc handle */ + rte_graph_feature_arc_t output_feature_arc; }; static struct ip4_rewrite_node_main *ip4_rewrite_nm; +static int port_to_next_index_diff; #define IP4_REWRITE_NODE_LAST_NEXT(ctx) \ (((struct ip4_rewrite_node_ctx *)ctx)->next_index) @@ -30,16 +43,175 @@ static struct ip4_rewrite_node_main *ip4_rewrite_nm; #define IP4_REWRITE_NODE_PRIV1_OFF(ctx) \ (((struct ip4_rewrite_node_ctx *)ctx)->mbuf_priv1_off) -static uint16_t -ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, - void **objs, uint16_t nb_objs) +#define IP4_REWRITE_NODE_FEAT_OFF(ctx) \ + (((struct ip4_rewrite_node_ctx *)ctx)->arc_dyn_off) + +#define IP4_REWRITE_NODE_OUTPUT_FEATURE_ARC(ctx) \ + (((struct ip4_rewrite_node_ctx *)ctx)->output_feature_arc) + +#define IP4_REWRITE_NODE_LAST_TX_IF(ctx) \ + (((struct ip4_rewrite_node_ctx *)ctx)->last_tx_if) + +static __rte_always_inline void +check_output_feature_arc_x1(struct rte_graph_feature_arc *arc, uint16_t *tx_if, + struct rte_mbuf *mbuf0, uint16_t *next0, + uint16_t *last_next_index, + rte_graph_feature_data_t *feature_data, const int dyn) +{ + struct rte_graph_feature_arc_mbuf_dynfields *d0 = NULL; + uint16_t port0; + + /* make sure packets are not being sent to pkt_drop node */ + if (unlikely(!(*next0))) + return; + + port0 = (*next0) - port_to_next_index_diff; + + /* get pointer to feature arc mbuf */ + d0 = rte_graph_feature_arc_mbuf_dynfields_get(mbuf0, dyn); + + /* Check if last packet's tx port not same as current */ + if (unlikely(*tx_if != port0)) { + if (unlikely(rte_graph_feature_data_first_feature_get(arc, port0, + &d0->feature_data))) { + *next0 = rte_graph_feature_data_edge_get(arc, d0->feature_data); + mbuf0->port = port0; + } + + *last_next_index = *next0; + *tx_if = port0; + *feature_data = d0->feature_data; + } else { + if (unlikely(rte_graph_feature_data_is_valid(*feature_data))) { + *next0 = *last_next_index; + mbuf0->port = *tx_if; + d0->feature_data = *feature_data; + } + } +} + +static __rte_always_inline void +check_output_feature_arc_x4(struct rte_graph_feature_arc *arc, uint16_t *tx_if, + struct rte_mbuf *mbuf0, struct rte_mbuf *mbuf1, + struct rte_mbuf *mbuf2, struct rte_mbuf *mbuf3, + uint16_t *next0, uint16_t *next1, uint16_t *next2, + uint16_t *next3, uint16_t *last_next_index, + rte_graph_feature_data_t *feature_data, const int dyn) +{ + struct rte_graph_feature_arc_mbuf_dynfields *d0 = NULL, *d1 = NULL, *d2 = NULL, *d3 = NULL; + uint16_t port0, port1, port2, port3; + uint16_t xor = 0; + + /* If no ip4_rewrite_set_next() is called yet */ + if (unlikely(!port_to_next_index_diff)) + return; + + /* get pointer to feature arc dyn field */ + d0 = rte_graph_feature_arc_mbuf_dynfields_get(mbuf0, dyn); + d1 = rte_graph_feature_arc_mbuf_dynfields_get(mbuf1, dyn); + d2 = rte_graph_feature_arc_mbuf_dynfields_get(mbuf2, dyn); + d3 = rte_graph_feature_arc_mbuf_dynfields_get(mbuf3, dyn); + + /* + * Check if all four packets are going to same next_index/port + */ + xor = ((*tx_if) + port_to_next_index_diff) ^ (*next0); + xor += (*next0) ^ (*next1); + xor += (*next1) ^ (*next2); + xor += (*next2) ^ (*next3); + + if (xor) { + /* packets tx ports are not same, check first feature for each mbuf + * make sure next0 != 0 which is pkt_drop + */ + port0 = (*next0) - port_to_next_index_diff; + port1 = (*next1) - port_to_next_index_diff; + port2 = (*next2) - port_to_next_index_diff; + port3 = (*next3) - port_to_next_index_diff; + if (unlikely((*next0 >= port_to_next_index_diff) && + rte_graph_feature_data_first_feature_get(arc, port0, + &d0->feature_data))) { + /* update next0 from feature arc */ + *next0 = rte_graph_feature_data_edge_get(arc, d0->feature_data); + mbuf0->port = port0; + + *tx_if = port0; + *last_next_index = *next0; + *feature_data = d0->feature_data; + } + + if (unlikely((*next1 >= port_to_next_index_diff) && + rte_graph_feature_data_first_feature_get(arc, port1, + &d1->feature_data))) { + port1 = (*next1) - port_to_next_index_diff; + *next1 = rte_graph_feature_data_edge_get(arc, d1->feature_data); + mbuf1->port = port1; + *tx_if = port1; + *last_next_index = *next1; + *feature_data = d1->feature_data; + } + + if (unlikely((*next2 >= port_to_next_index_diff) && + rte_graph_feature_data_first_feature_get(arc, port2, + &d2->feature_data))) { + port2 = (*next2) - port_to_next_index_diff; + *next2 = rte_graph_feature_data_edge_get(arc, d2->feature_data); + mbuf2->port = port2; + *tx_if = port2; + *last_next_index = *next2; + *feature_data = d2->feature_data; + } + + if (unlikely((*next3 >= port_to_next_index_diff) && + rte_graph_feature_data_first_feature_get(arc, port3, + &d3->feature_data))) { + port3 = (*next3) - port_to_next_index_diff; + *next3 = rte_graph_feature_data_edge_get(arc, d3->feature_data); + mbuf3->port = port3; + *tx_if = port3; + *last_next_index = *next3; + *feature_data = d3->feature_data; + } + } else { + /* All packets are same as last tx port. Check if feature enabled + * on last packet is valid or not. If invalid no need to + * change any next[0-3] + * Also check packet is not being sent to pkt_drop node + */ + if (unlikely(rte_graph_feature_data_is_valid(*feature_data) && + (*next0 != 0))) { + *next0 = *last_next_index; + *next1 = *last_next_index; + *next2 = *last_next_index; + *next3 = *last_next_index; + + d0->feature_data = *feature_data; + d1->feature_data = *feature_data; + d2->feature_data = *feature_data; + d3->feature_data = *feature_data; + + mbuf0->port = *tx_if; + mbuf1->port = *tx_if; + mbuf2->port = *tx_if; + mbuf3->port = *tx_if; + } + } +} + +static __rte_always_inline uint16_t +__ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, + void **objs, uint16_t nb_objs, + const int dyn, const int check_enabled_features, + const int feat_dyn, + struct rte_graph_feature_arc *out_feature_arc) { struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts; struct ip4_rewrite_nh_header *nh = ip4_rewrite_nm->nh; - const int dyn = IP4_REWRITE_NODE_PRIV1_OFF(node->ctx); uint16_t next0, next1, next2, next3, next_index; struct rte_ipv4_hdr *ip0, *ip1, *ip2, *ip3; uint16_t n_left_from, held = 0, last_spec = 0; + rte_graph_feature_data_t feature_data; + uint16_t last_tx_if, last_next_index; void *d0, *d1, *d2, *d3; void **to_next, **from; rte_xmm_t priv01; @@ -57,6 +229,27 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, for (i = 0; i < 4 && i < n_left_from; i++) rte_prefetch0(pkts[i]); + if (check_enabled_features) { + rte_graph_feature_arc_prefetch(out_feature_arc); + + last_tx_if = IP4_REWRITE_NODE_LAST_TX_IF(node->ctx); + + /* If feature is enabled on last_tx_if, prefetch data + * corresponding to first feature + */ + if (unlikely(rte_graph_feature_data_first_feature_get(out_feature_arc, + last_tx_if, + &feature_data))) + rte_graph_feature_arc_feature_data_prefetch(out_feature_arc, + feature_data); + + /* Reset last_tx_if and last_next_index to call feature arc APIs + * for initial packets in every node loop + */ + last_tx_if = UINT16_MAX; + last_next_index = UINT16_MAX; + } + /* Get stream for the speculated next node */ to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs); /* Update Ethernet header of pkts */ @@ -78,6 +271,7 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, pkts += 4; n_left_from -= 4; + priv01.u64[0] = node_mbuf_priv1(mbuf0, dyn)->u; priv01.u64[1] = node_mbuf_priv1(mbuf1, dyn)->u; priv23.u64[0] = node_mbuf_priv1(mbuf2, dyn)->u; @@ -132,6 +326,16 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, ip3->time_to_live = priv23.u16[5] - 1; ip3->hdr_checksum = priv23.u16[6] + priv23.u16[7]; + /* Once all mbufs are updated with next hop data. + * check if any feature is enabled to override + * next edges + */ + if (check_enabled_features) + check_output_feature_arc_x4(out_feature_arc, &last_tx_if, + mbuf0, mbuf1, mbuf2, mbuf3, + &next0, &next1, &next2, &next3, + &last_next_index, &feature_data, feat_dyn); + /* Enqueue four to next node */ rte_edge_t fix_spec = ((next_index == next0) && (next0 == next1) && @@ -225,6 +429,11 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, ip0->hdr_checksum = chksum; ip0->time_to_live = node_mbuf_priv1(mbuf0, dyn)->ttl - 1; + if (check_enabled_features) + check_output_feature_arc_x1(out_feature_arc, &last_tx_if, + mbuf0, &next0, &last_next_index, + &feature_data, feat_dyn); + if (unlikely(next_index ^ next0)) { /* Copy things successfully speculated till now */ rte_memcpy(to_next, from, last_spec * sizeof(from[0])); @@ -252,12 +461,49 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, /* Save the last next used */ IP4_REWRITE_NODE_LAST_NEXT(node->ctx) = next_index; + if (check_enabled_features) + IP4_REWRITE_NODE_LAST_TX_IF(node->ctx) = last_tx_if; + return nb_objs; } +static uint16_t +ip4_rewrite_feature_node_process(struct rte_graph *graph, struct rte_node *node, + void **objs, uint16_t nb_objs) +{ + struct rte_graph_feature_arc *arc = NULL; + rte_graph_feature_arc_get(IP4_REWRITE_NODE_OUTPUT_FEATURE_ARC(node->ctx)); + const int feat_dyn = IP4_REWRITE_NODE_FEAT_OFF(node->ctx); + const int dyn = IP4_REWRITE_NODE_PRIV1_OFF(node->ctx); + + arc = rte_graph_feature_arc_get(IP4_REWRITE_NODE_OUTPUT_FEATURE_ARC(node->ctx)); + if (unlikely(arc && rte_graph_feature_arc_is_any_feature_enabled(arc))) + return __ip4_rewrite_node_process(graph, node, objs, nb_objs, dyn, + 1 /* check features */, feat_dyn, arc); + + return __ip4_rewrite_node_process(graph, node, objs, nb_objs, dyn, + 0/* don't check features*/, + 0 /* don't care */, + NULL /* don't care*/); +} + +static uint16_t +ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, + void **objs, uint16_t nb_objs) +{ + const int dyn = IP4_REWRITE_NODE_PRIV1_OFF(node->ctx); + + return __ip4_rewrite_node_process(graph, node, objs, nb_objs, dyn, + 0/* don't check features*/, + 0 /* don't care */, + NULL/* don't care */); +} + + static int ip4_rewrite_node_init(const struct rte_graph *graph, struct rte_node *node) { + rte_graph_feature_arc_t feature_arc; static bool init_once; RTE_SET_USED(graph); @@ -268,9 +514,27 @@ ip4_rewrite_node_init(const struct rte_graph *graph, struct rte_node *node) &node_mbuf_priv1_dynfield_desc); if (node_mbuf_priv1_dynfield_offset < 0) return -rte_errno; + + /* Create ipv4-output feature arc, if not created + */ + if (rte_graph_feature_arc_lookup_by_name(RTE_IP4_OUTPUT_FEATURE_ARC_NAME, + &feature_arc) < 0) { + node_err("ip4_rewrite", "Feature arc \"%s\" not found", + RTE_IP4_OUTPUT_FEATURE_ARC_NAME); + } else { + node_err("ip4_rewrite", "Feature arc \"%s\" found", + RTE_IP4_OUTPUT_FEATURE_ARC_NAME); + } + init_once = true; } IP4_REWRITE_NODE_PRIV1_OFF(node->ctx) = node_mbuf_priv1_dynfield_offset; + IP4_REWRITE_NODE_FEAT_OFF(node->ctx) = rte_graph_feature_arc_mbuf_dynfield_offset_get(); + IP4_REWRITE_NODE_OUTPUT_FEATURE_ARC(node->ctx) = feature_arc; + + /* By default, set cached next node to pkt_drop */ + IP4_REWRITE_NODE_LAST_NEXT(node->ctx) = 0; + IP4_REWRITE_NODE_LAST_TX_IF(node->ctx) = 0; node_dbg("ip4_rewrite", "Initialized ip4_rewrite node initialized"); @@ -280,6 +544,8 @@ ip4_rewrite_node_init(const struct rte_graph *graph, struct rte_node *node) int ip4_rewrite_set_next(uint16_t port_id, uint16_t next_index) { + static int once; + if (ip4_rewrite_nm == NULL) { ip4_rewrite_nm = rte_zmalloc( "ip4_rewrite", sizeof(struct ip4_rewrite_node_main), @@ -287,6 +553,10 @@ ip4_rewrite_set_next(uint16_t port_id, uint16_t next_index) if (ip4_rewrite_nm == NULL) return -ENOMEM; } + if (!once) { + port_to_next_index_diff = next_index - port_id; + once = 1; + } ip4_rewrite_nm->next_index[port_id] = next_index; return 0; @@ -345,3 +615,23 @@ ip4_rewrite_node_get(void) } RTE_NODE_REGISTER(ip4_rewrite_node); + +/* IP4 output arc */ +static struct rte_graph_feature_arc_register ip4_output_arc = { + .arc_name = RTE_IP4_OUTPUT_FEATURE_ARC_NAME, + + /* This arc works on all ethdevs */ + .max_indexes = RTE_IP4_OUTPUT_ARC_INDEXES, + + .start_node = &ip4_rewrite_node, + + /* overwrites start_node->process() function with following only if + * application calls rte_graph_feature_arc_init() + */ + .start_node_feature_process_fn = ip4_rewrite_feature_node_process, + + /* end feature node of an arc*/ + .end_feature = &if_tx_feature, +}; + +RTE_GRAPH_FEATURE_ARC_REGISTER(ip4_output_arc); diff --git a/lib/node/meson.build b/lib/node/meson.build index 0bed97a96c..11e03d9ef6 100644 --- a/lib/node/meson.build +++ b/lib/node/meson.build @@ -24,6 +24,7 @@ sources = files( 'pkt_cls.c', 'pkt_drop.c', 'udp4_input.c', + 'interface_tx_feature.c' ) headers = files( 'rte_node_eth_api.h', diff --git a/lib/node/node_private.h b/lib/node/node_private.h index 4fafab19be..9488df225c 100644 --- a/lib/node/node_private.h +++ b/lib/node/node_private.h @@ -13,6 +13,7 @@ #include #include +#include extern int rte_node_logtype; #define RTE_LOGTYPE_NODE rte_node_logtype diff --git a/lib/node/rte_node_ip4_api.h b/lib/node/rte_node_ip4_api.h index 950751a525..8cfae14b46 100644 --- a/lib/node/rte_node_ip4_api.h +++ b/lib/node/rte_node_ip4_api.h @@ -24,6 +24,10 @@ extern "C" { #endif +/** IP4 output arc */ +#define RTE_IP4_OUTPUT_FEATURE_ARC_NAME "rte_ip4_output_arc" +#define RTE_IP4_OUTPUT_END_FEATURE_NAME "rte_if_tx_feature" + /** * IP4 lookup next nodes. */ -- 2.43.0