From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3A127A04DD; Wed, 28 Oct 2020 11:28:24 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 06F31C9E0; Wed, 28 Oct 2020 11:27:14 +0100 (CET) Received: from new1-smtp.messagingengine.com (new1-smtp.messagingengine.com [66.111.4.221]) by dpdk.org (Postfix) with ESMTP id 7CC36C9D4 for ; Wed, 28 Oct 2020 11:27:10 +0100 (CET) Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailnew.nyi.internal (Postfix) with ESMTP id 2E40B580388; Wed, 28 Oct 2020 06:27:09 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Wed, 28 Oct 2020 06:27:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; s=fm2; bh=b2/UgouepwNuH akdJkmPqZDAY7jia25Ie/3cISWNZ/4=; b=NEmu+9W4Hi8cidOKzp4DwszBQC/qV DOvx4IuXfJN2dIoTS50kxVWUYXsPWPLVB+rogpdgX5cXly41QmGAu8G7rS2gHTur pYiuvSZcnZHvcQEKfQifF2riCOTB5VmuQ+lQ3ukyFFcUGedY/Alq3YH7g7p0Dvxy dJmai5KQmWpyG8KaLkvO8FNNh6T5J9WGzzvGOxMvDlgpNFbzXKtf3luxJFTLsLDs mTFBcgYw1LYBGG2MWKAQGyOe6IJ73b0v9P+bdXUBRfPfPWE7unUXOKCOhy4qq+1B qXw3n26gQQ8MViCCWk0BLk3/OHQdxPfEaaRtYrwC0RhMfcNYtslLH9QRw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:subject:to :x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; bh=b2/UgouepwNuHakdJkmPqZDAY7jia25Ie/3cISWNZ/4=; b=GL6FkAJ6 5BOiH4eLh20yx3gexG5grYvgXrGtPHFyJvO4X05o7y9I9qqcYrr3ZBSwwIe63Ekq CWK82rtQUR/hq+35JKZNgxBS47Hd7xHec4oCDTpqD0RWzHX9kmyWGERk1ah6mqoP agwlE5sJlbJezX4gYOnxOFg51CgjU672/lP6NdQhX5Qk4vh4SSoF79K05KBcxufu LzgnQKmSYs9NnCdSrSi6oXuci/UjVVP2FP5bFsAx/r6ZMjsPzYax56MbHYoyTtHB Bwbvz6VuhAR+OU889rTkga9SmnGVr5nh+Trh/3H8nUX8p7QqHoVhBmtARkelYA3t iOZEJdEFNeKlVA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrledugdduhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffkofgjfhgggfestdekredtredttdenucfhrhhomhepvfhhohhmrghs ucfoohhnjhgrlhhonhcuoehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucggtf frrghtthgvrhhnpedvhefgiedvjeegtdevheefhfetleefgfeivefgffevfeejgedtgfeu tdehtdegveenucfkphepjeejrddufeegrddvtdefrddukeegnecuvehluhhsthgvrhfuih iivgepfeenucfrrghrrghmpehmrghilhhfrhhomhepthhhohhmrghssehmohhnjhgrlhho nhdrnhgvth X-ME-Proxy: Received: from xps.monjalon.net (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id 603C23280059; Wed, 28 Oct 2020 06:27:07 -0400 (EDT) From: Thomas Monjalon To: dev@dpdk.org Cc: ferruh.yigit@intel.com, david.marchand@redhat.com, bruce.richardson@intel.com, olivier.matz@6wind.com, andrew.rybchenko@oktetlabs.ru, akhil.goyal@nxp.com, Nithin Dabilpuram , Pavan Nikhilesh , Jerin Jacob , Ruifeng Wang , Konstantin Ananyev Date: Wed, 28 Oct 2020 11:26:29 +0100 Message-Id: <20201028102640.3191964-5-thomas@monjalon.net> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20201028102640.3191964-1-thomas@monjalon.net> References: <20201026052105.1561859-1-thomas@monjalon.net> <20201028102640.3191964-1-thomas@monjalon.net> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-dev] [PATCH v4 04/15] node: switch IPv4 metadata to dynamic mbuf field X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" From: Nithin Dabilpuram The node_mbuf_priv1 was stored in the deprecated mbuf field udata64. It is moved to a dynamic field in order to allow removal of udata64. Signed-off-by: Thomas Monjalon Signed-off-by: Nithin Dabilpuram --- lib/librte_node/ip4_lookup.c | 40 ++++++++++++++++++------- lib/librte_node/ip4_lookup_neon.h | 20 ++++++------- lib/librte_node/ip4_lookup_sse.h | 36 +++++++++++------------ lib/librte_node/ip4_rewrite.c | 49 +++++++++++++++++++++++-------- lib/librte_node/node_private.h | 13 ++++++-- 5 files changed, 103 insertions(+), 55 deletions(-) diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c index 8835aab9dd..d083a725fc 100644 --- a/lib/librte_node/ip4_lookup.c +++ b/lib/librte_node/ip4_lookup.c @@ -29,8 +29,23 @@ struct ip4_lookup_node_main { struct rte_lpm *lpm_tbl[RTE_MAX_NUMA_NODES]; }; +struct ip4_lookup_node_ctx { + /* Socket's LPM table */ + struct rte_lpm *lpm; + /* Dynamic offset to mbuf priv1 */ + int mbuf_priv1_off; +}; + +int node_mbuf_priv1_dynfield_offset = -1; + static struct ip4_lookup_node_main ip4_lookup_nm; +#define IP4_LOOKUP_NODE_LPM(ctx) \ + (((struct ip4_lookup_node_ctx *)ctx)->lpm) + +#define IP4_LOOKUP_NODE_PRIV1_OFF(ctx) \ + (((struct ip4_lookup_node_ctx *)ctx)->mbuf_priv1_off) + #if defined(__ARM_NEON) #include "ip4_lookup_neon.h" #elif defined(RTE_ARCH_X86) @@ -41,12 +56,13 @@ static uint16_t ip4_lookup_node_process_scalar(struct rte_graph *graph, struct rte_node *node, void **objs, uint16_t nb_objs) { + struct rte_lpm *lpm = IP4_LOOKUP_NODE_LPM(node->ctx); + const int dyn = IP4_LOOKUP_NODE_PRIV1_OFF(node->ctx); struct rte_ipv4_hdr *ipv4_hdr; void **to_next, **from; uint16_t last_spec = 0; struct rte_mbuf *mbuf; rte_edge_t next_index; - struct rte_lpm *lpm; uint16_t held = 0; uint32_t drop_nh; int i, rc; @@ -55,9 +71,6 @@ ip4_lookup_node_process_scalar(struct rte_graph *graph, struct rte_node *node, next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE; /* Drop node */ drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16; - - /* Get socket specific LPM from ctx */ - lpm = *((struct rte_lpm **)node->ctx); from = objs; /* Get stream for the speculated next node */ @@ -72,14 +85,14 @@ ip4_lookup_node_process_scalar(struct rte_graph *graph, struct rte_node *node, ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf, struct rte_ipv4_hdr *, sizeof(struct rte_ether_hdr)); /* Extract cksum, ttl as ipv4 hdr is in cache */ - node_mbuf_priv1(mbuf)->cksum = ipv4_hdr->hdr_checksum; - node_mbuf_priv1(mbuf)->ttl = ipv4_hdr->time_to_live; + node_mbuf_priv1(mbuf, dyn)->cksum = ipv4_hdr->hdr_checksum; + node_mbuf_priv1(mbuf, dyn)->ttl = ipv4_hdr->time_to_live; rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr), &next_hop); next_hop = (rc == 0) ? next_hop : drop_nh; - node_mbuf_priv1(mbuf)->nh = (uint16_t)next_hop; + node_mbuf_priv1(mbuf, dyn)->nh = (uint16_t)next_hop; next_hop = next_hop >> 16; next = (uint16_t)next_hop; @@ -169,15 +182,19 @@ setup_lpm(struct ip4_lookup_node_main *nm, int socket) static int ip4_lookup_node_init(const struct rte_graph *graph, struct rte_node *node) { - struct rte_lpm **lpm_p = (struct rte_lpm **)&node->ctx; uint16_t socket, lcore_id; static uint8_t init_once; int rc; RTE_SET_USED(graph); - RTE_SET_USED(node); + RTE_BUILD_BUG_ON(sizeof(struct ip4_lookup_node_ctx) > RTE_NODE_CTX_SZ); if (!init_once) { + node_mbuf_priv1_dynfield_offset = rte_mbuf_dynfield_register( + &node_mbuf_priv1_dynfield_desc); + if (node_mbuf_priv1_dynfield_offset < 0) + return -rte_errno; + /* Setup LPM tables for all sockets */ RTE_LCORE_FOREACH(lcore_id) { @@ -192,7 +209,10 @@ ip4_lookup_node_init(const struct rte_graph *graph, struct rte_node *node) } init_once = 1; } - *lpm_p = ip4_lookup_nm.lpm_tbl[graph->socket]; + + /* Update socket's LPM and mbuf dyn priv1 offset in node ctx */ + IP4_LOOKUP_NODE_LPM(node->ctx) = ip4_lookup_nm.lpm_tbl[graph->socket]; + IP4_LOOKUP_NODE_PRIV1_OFF(node->ctx) = node_mbuf_priv1_dynfield_offset; #if defined(__ARM_NEON) || defined(RTE_ARCH_X86) if (rte_vect_get_max_simd_bitwidth() >= RTE_VECT_SIMD_128) diff --git a/lib/librte_node/ip4_lookup_neon.h b/lib/librte_node/ip4_lookup_neon.h index 0ad2763b82..d5c8da3719 100644 --- a/lib/librte_node/ip4_lookup_neon.h +++ b/lib/librte_node/ip4_lookup_neon.h @@ -11,12 +11,13 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, void **objs, uint16_t nb_objs) { struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts; + struct rte_lpm *lpm = IP4_LOOKUP_NODE_LPM(node->ctx); + const int dyn = IP4_LOOKUP_NODE_PRIV1_OFF(node->ctx); struct rte_ipv4_hdr *ipv4_hdr; void **to_next, **from; uint16_t last_spec = 0; rte_edge_t next_index; uint16_t n_left_from; - struct rte_lpm *lpm; uint16_t held = 0; uint32_t drop_nh; rte_xmm_t result; @@ -30,9 +31,6 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, /* Drop node */ drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16; - /* Get socket specific LPM from ctx */ - lpm = *((struct rte_lpm **)node->ctx); - pkts = (struct rte_mbuf **)objs; from = objs; n_left_from = nb_objs; @@ -119,10 +117,10 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, priv23.u16[0] = result.u16[4]; priv23.u16[4] = result.u16[6]; - node_mbuf_priv1(mbuf0)->u = priv01.u64[0]; - node_mbuf_priv1(mbuf1)->u = priv01.u64[1]; - node_mbuf_priv1(mbuf2)->u = priv23.u64[0]; - node_mbuf_priv1(mbuf3)->u = priv23.u64[1]; + node_mbuf_priv1(mbuf0, dyn)->u = priv01.u64[0]; + node_mbuf_priv1(mbuf1, dyn)->u = priv01.u64[1]; + node_mbuf_priv1(mbuf2, dyn)->u = priv23.u64[0]; + node_mbuf_priv1(mbuf3, dyn)->u = priv23.u64[1]; /* Enqueue four to next node */ rte_edge_t fix_spec = ((next_index == result.u16[1]) && @@ -197,14 +195,14 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *, sizeof(struct rte_ether_hdr)); /* Extract cksum, ttl as ipv4 hdr is in cache */ - node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum; - node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live; + node_mbuf_priv1(mbuf0, dyn)->cksum = ipv4_hdr->hdr_checksum; + node_mbuf_priv1(mbuf0, dyn)->ttl = ipv4_hdr->time_to_live; rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr), &next_hop); next_hop = (rc == 0) ? next_hop : drop_nh; - node_mbuf_priv1(mbuf0)->nh = (uint16_t)next_hop; + node_mbuf_priv1(mbuf0, dyn)->nh = (uint16_t)next_hop; next_hop = next_hop >> 16; next0 = (uint16_t)next_hop; diff --git a/lib/librte_node/ip4_lookup_sse.h b/lib/librte_node/ip4_lookup_sse.h index 264c986071..74dbf97533 100644 --- a/lib/librte_node/ip4_lookup_sse.h +++ b/lib/librte_node/ip4_lookup_sse.h @@ -11,13 +11,14 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, void **objs, uint16_t nb_objs) { struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts; + struct rte_lpm *lpm = IP4_LOOKUP_NODE_LPM(node->ctx); + const int dyn = IP4_LOOKUP_NODE_PRIV1_OFF(node->ctx); rte_edge_t next0, next1, next2, next3, next_index; struct rte_ipv4_hdr *ipv4_hdr; uint32_t ip0, ip1, ip2, ip3; void **to_next, **from; uint16_t last_spec = 0; uint16_t n_left_from; - struct rte_lpm *lpm; uint16_t held = 0; uint32_t drop_nh; rte_xmm_t dst; @@ -29,9 +30,6 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, /* Drop node */ drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16; - /* Get socket specific LPM from ctx */ - lpm = *((struct rte_lpm **)node->ctx); - pkts = (struct rte_mbuf **)objs; from = objs; n_left_from = nb_objs; @@ -78,24 +76,24 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, sizeof(struct rte_ether_hdr)); ip0 = ipv4_hdr->dst_addr; /* Extract cksum, ttl as ipv4 hdr is in cache */ - node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum; - node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live; + node_mbuf_priv1(mbuf0, dyn)->cksum = ipv4_hdr->hdr_checksum; + node_mbuf_priv1(mbuf0, dyn)->ttl = ipv4_hdr->time_to_live; /* Extract DIP of mbuf1 */ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf1, struct rte_ipv4_hdr *, sizeof(struct rte_ether_hdr)); ip1 = ipv4_hdr->dst_addr; /* Extract cksum, ttl as ipv4 hdr is in cache */ - node_mbuf_priv1(mbuf1)->cksum = ipv4_hdr->hdr_checksum; - node_mbuf_priv1(mbuf1)->ttl = ipv4_hdr->time_to_live; + node_mbuf_priv1(mbuf1, dyn)->cksum = ipv4_hdr->hdr_checksum; + node_mbuf_priv1(mbuf1, dyn)->ttl = ipv4_hdr->time_to_live; /* Extract DIP of mbuf2 */ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf2, struct rte_ipv4_hdr *, sizeof(struct rte_ether_hdr)); ip2 = ipv4_hdr->dst_addr; /* Extract cksum, ttl as ipv4 hdr is in cache */ - node_mbuf_priv1(mbuf2)->cksum = ipv4_hdr->hdr_checksum; - node_mbuf_priv1(mbuf2)->ttl = ipv4_hdr->time_to_live; + node_mbuf_priv1(mbuf2, dyn)->cksum = ipv4_hdr->hdr_checksum; + node_mbuf_priv1(mbuf2, dyn)->ttl = ipv4_hdr->time_to_live; /* Extract DIP of mbuf3 */ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf3, struct rte_ipv4_hdr *, @@ -111,23 +109,23 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, dip = _mm_shuffle_epi8(dip, bswap_mask); /* Extract cksum, ttl as ipv4 hdr is in cache */ - node_mbuf_priv1(mbuf3)->cksum = ipv4_hdr->hdr_checksum; - node_mbuf_priv1(mbuf3)->ttl = ipv4_hdr->time_to_live; + node_mbuf_priv1(mbuf3, dyn)->cksum = ipv4_hdr->hdr_checksum; + node_mbuf_priv1(mbuf3, dyn)->ttl = ipv4_hdr->time_to_live; /* Perform LPM lookup to get NH and next node */ rte_lpm_lookupx4(lpm, dip, dst.u32, drop_nh); /* Extract next node id and NH */ - node_mbuf_priv1(mbuf0)->nh = dst.u32[0] & 0xFFFF; + node_mbuf_priv1(mbuf0, dyn)->nh = dst.u32[0] & 0xFFFF; next0 = (dst.u32[0] >> 16); - node_mbuf_priv1(mbuf1)->nh = dst.u32[1] & 0xFFFF; + node_mbuf_priv1(mbuf1, dyn)->nh = dst.u32[1] & 0xFFFF; next1 = (dst.u32[1] >> 16); - node_mbuf_priv1(mbuf2)->nh = dst.u32[2] & 0xFFFF; + node_mbuf_priv1(mbuf2, dyn)->nh = dst.u32[2] & 0xFFFF; next2 = (dst.u32[2] >> 16); - node_mbuf_priv1(mbuf3)->nh = dst.u32[3] & 0xFFFF; + node_mbuf_priv1(mbuf3, dyn)->nh = dst.u32[3] & 0xFFFF; next3 = (dst.u32[3] >> 16); /* Enqueue four to next node */ @@ -202,14 +200,14 @@ ip4_lookup_node_process_vec(struct rte_graph *graph, struct rte_node *node, ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *, sizeof(struct rte_ether_hdr)); /* Extract cksum, ttl as ipv4 hdr is in cache */ - node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum; - node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live; + node_mbuf_priv1(mbuf0, dyn)->cksum = ipv4_hdr->hdr_checksum; + node_mbuf_priv1(mbuf0, dyn)->ttl = ipv4_hdr->time_to_live; rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr), &next_hop); next_hop = (rc == 0) ? next_hop : drop_nh; - node_mbuf_priv1(mbuf0)->nh = next_hop & 0xFFFF; + node_mbuf_priv1(mbuf0, dyn)->nh = next_hop & 0xFFFF; next0 = (next_hop >> 16); if (unlikely(next_index ^ next0)) { diff --git a/lib/librte_node/ip4_rewrite.c b/lib/librte_node/ip4_rewrite.c index bb7f671b5c..99ecb457ff 100644 --- a/lib/librte_node/ip4_rewrite.c +++ b/lib/librte_node/ip4_rewrite.c @@ -19,14 +19,28 @@ #include "ip4_rewrite_priv.h" #include "node_private.h" +struct ip4_rewrite_node_ctx { + /* Dynamic offset to mbuf priv1 */ + int mbuf_priv1_off; + /* Cached next index */ + uint16_t next_index; +}; + static struct ip4_rewrite_node_main *ip4_rewrite_nm; +#define IP4_REWRITE_NODE_LAST_NEXT(ctx) \ + (((struct ip4_rewrite_node_ctx *)ctx)->next_index) + +#define IP4_REWRITE_NODE_PRIV1_OFF(ctx) \ + (((struct ip4_rewrite_node_ctx *)ctx)->mbuf_priv1_off) + static uint16_t ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, void **objs, uint16_t nb_objs) { struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts; struct ip4_rewrite_nh_header *nh = ip4_rewrite_nm->nh; + const int dyn = IP4_REWRITE_NODE_PRIV1_OFF(node->ctx); uint16_t next0, next1, next2, next3, next_index; struct rte_ipv4_hdr *ip0, *ip1, *ip2, *ip3; uint16_t n_left_from, held = 0, last_spec = 0; @@ -37,7 +51,7 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, int i; /* Speculative next as last next */ - next_index = *(uint16_t *)node->ctx; + next_index = IP4_REWRITE_NODE_LAST_NEXT(node->ctx); rte_prefetch0(nh); pkts = (struct rte_mbuf **)objs; @@ -68,10 +82,10 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, pkts += 4; n_left_from -= 4; - priv01.u64[0] = node_mbuf_priv1(mbuf0)->u; - priv01.u64[1] = node_mbuf_priv1(mbuf1)->u; - priv23.u64[0] = node_mbuf_priv1(mbuf2)->u; - priv23.u64[1] = node_mbuf_priv1(mbuf3)->u; + priv01.u64[0] = node_mbuf_priv1(mbuf0, dyn)->u; + priv01.u64[1] = node_mbuf_priv1(mbuf1, dyn)->u; + priv23.u64[0] = node_mbuf_priv1(mbuf2, dyn)->u; + priv23.u64[1] = node_mbuf_priv1(mbuf3, dyn)->u; /* Increment checksum by one. */ priv01.u32[1] += rte_cpu_to_be_16(0x0100); @@ -203,17 +217,17 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, n_left_from -= 1; d0 = rte_pktmbuf_mtod(mbuf0, void *); - rte_memcpy(d0, nh[node_mbuf_priv1(mbuf0)->nh].rewrite_data, - nh[node_mbuf_priv1(mbuf0)->nh].rewrite_len); + rte_memcpy(d0, nh[node_mbuf_priv1(mbuf0, dyn)->nh].rewrite_data, + nh[node_mbuf_priv1(mbuf0, dyn)->nh].rewrite_len); - next0 = nh[node_mbuf_priv1(mbuf0)->nh].tx_node; + next0 = nh[node_mbuf_priv1(mbuf0, dyn)->nh].tx_node; ip0 = (struct rte_ipv4_hdr *)((uint8_t *)d0 + sizeof(struct rte_ether_hdr)); - chksum = node_mbuf_priv1(mbuf0)->cksum + + chksum = node_mbuf_priv1(mbuf0, dyn)->cksum + rte_cpu_to_be_16(0x0100); chksum += chksum >= 0xffff; ip0->hdr_checksum = chksum; - ip0->time_to_live = node_mbuf_priv1(mbuf0)->ttl - 1; + ip0->time_to_live = node_mbuf_priv1(mbuf0, dyn)->ttl - 1; if (unlikely(next_index ^ next0)) { /* Copy things successfully speculated till now */ @@ -240,7 +254,7 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, rte_memcpy(to_next, from, last_spec * sizeof(from[0])); rte_node_next_stream_put(graph, node, next_index, held); /* Save the last next used */ - *(uint16_t *)node->ctx = next_index; + IP4_REWRITE_NODE_LAST_NEXT(node->ctx) = next_index; return nb_objs; } @@ -248,9 +262,20 @@ ip4_rewrite_node_process(struct rte_graph *graph, struct rte_node *node, static int ip4_rewrite_node_init(const struct rte_graph *graph, struct rte_node *node) { + static bool init_once; RTE_SET_USED(graph); - RTE_SET_USED(node); + RTE_BUILD_BUG_ON(sizeof(struct ip4_rewrite_node_ctx) > RTE_NODE_CTX_SZ); + + if (!init_once) { + node_mbuf_priv1_dynfield_offset = rte_mbuf_dynfield_register( + &node_mbuf_priv1_dynfield_desc); + if (node_mbuf_priv1_dynfield_offset < 0) + return -rte_errno; + init_once = true; + } + IP4_REWRITE_NODE_PRIV1_OFF(node->ctx) = node_mbuf_priv1_dynfield_offset; + node_dbg("ip4_rewrite", "Initialized ip4_rewrite node initialized"); return 0; diff --git a/lib/librte_node/node_private.h b/lib/librte_node/node_private.h index ab7941c12b..8c73d5dc10 100644 --- a/lib/librte_node/node_private.h +++ b/lib/librte_node/node_private.h @@ -8,6 +8,7 @@ #include #include #include +#include extern int rte_node_logtype; #define NODE_LOG(level, node_name, ...) \ @@ -21,7 +22,6 @@ extern int rte_node_logtype; #define node_dbg(node_name, ...) NODE_LOG(DEBUG, node_name, __VA_ARGS__) /** - * * Node mbuf private data to store next hop, ttl and checksum. */ struct node_mbuf_priv1 { @@ -37,6 +37,13 @@ struct node_mbuf_priv1 { }; }; +static const struct rte_mbuf_dynfield node_mbuf_priv1_dynfield_desc = { + .name = "rte_node_dynfield_priv1", + .size = sizeof(struct node_mbuf_priv1), + .align = __alignof__(struct node_mbuf_priv1), +}; +extern int node_mbuf_priv1_dynfield_offset; + /** * Node mbuf private area 2. */ @@ -58,9 +65,9 @@ struct node_mbuf_priv2 { * Pointer to the mbuf_priv1. */ static __rte_always_inline struct node_mbuf_priv1 * -node_mbuf_priv1(struct rte_mbuf *m) +node_mbuf_priv1(struct rte_mbuf *m, const int offset) { - return (struct node_mbuf_priv1 *)&m->udata64; + return RTE_MBUF_DYNFIELD(m, offset, struct node_mbuf_priv1 *); } /** -- 2.28.0