From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
by inbox.dpdk.org (Postfix) with ESMTP id C8EC5A0562;
Tue, 31 Mar 2020 21:33:07 +0200 (CEST)
Received: from [92.243.14.124] (localhost [127.0.0.1])
by dpdk.org (Postfix) with ESMTP id 997521C247;
Tue, 31 Mar 2020 21:30:57 +0200 (CEST)
Received: from mx0b-0016f401.pphosted.com (mx0a-0016f401.pphosted.com
[67.231.148.174]) by dpdk.org (Postfix) with ESMTP id C5C7D1C1A9
for <dev@dpdk.org>; Tue, 31 Mar 2020 21:30:54 +0200 (CEST)
Received: from pps.filterd (m0045849.ppops.net [127.0.0.1])
by mx0a-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id
02VJTNcb030443; Tue, 31 Mar 2020 12:30:52 -0700
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com;
h=from : to : cc :
subject : date : message-id : in-reply-to : references : mime-version :
content-transfer-encoding : content-type; s=pfpt0818;
bh=nBt79rs1ihZx4VetWTqmBQh/cVZYrGvK2Arcpw9BSSM=;
b=swT1hc/+4Uex8ejz/Lf4O67Ykf7ispaPsYhqhra1Ex87t8lQx6q44kQgmslJPxLUf8Lt
WwSJrPGBMmitCC8fYACYW1aloc3Illl9M6hRwWVZj0jpqzI19UC/xndYEjddu3fBufNP
F5m7Qb27mgGxoGE6SDR6QAFEyMgG1ZP0hDcVE78yXcjVNBgRvrK/xi03rXksWBaO0pXv
Z4wqHmYBLcsgGCBbR9jup9mBLs8bkgxvVsWqVMDa8zxDAcciG8lquByzGdIxfqa54LjE
tjHo1Zdb4kgg3a8V2FiVRsDoVFIqTnTdeasvGlMWhzkJPjtNRk1RRUpVvYtd9F5DLwo9 6g==
Received: from sc-exch02.marvell.com ([199.233.58.182])
by mx0a-0016f401.pphosted.com with ESMTP id 304855h77g-1
(version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT);
Tue, 31 Mar 2020 12:30:52 -0700
Received: from DC5-EXCH02.marvell.com (10.69.176.39) by SC-EXCH02.marvell.com
(10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1497.2;
Tue, 31 Mar 2020 12:30:51 -0700
Received: from SC-EXCH01.marvell.com (10.93.176.81) by DC5-EXCH02.marvell.com
(10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2;
Tue, 31 Mar 2020 12:30:50 -0700
Received: from maili.marvell.com (10.93.176.43) by SC-EXCH01.marvell.com
(10.93.176.81) with Microsoft SMTP Server id 15.0.1497.2 via Frontend
Transport; Tue, 31 Mar 2020 12:30:49 -0700
Received: from jerin-lab.marvell.com (jerin-lab.marvell.com [10.28.34.14])
by maili.marvell.com (Postfix) with ESMTP id 9955A3F7040;
Tue, 31 Mar 2020 12:30:47 -0700 (PDT)
From: <jerinj@marvell.com>
To: Nithin Dabilpuram <ndabilpuram@marvell.com>, Pavan Nikhilesh
<pbhagavatula@marvell.com>
CC: <dev@dpdk.org>, <thomas@monjalon.net>, <david.marchand@redhat.com>,
<mdr@ashroe.eu>, <mattias.ronnblom@ericsson.com>, <kirankumark@marvell.com>
Date: Wed, 1 Apr 2020 00:59:36 +0530
Message-ID: <20200331192945.2466880-21-jerinj@marvell.com>
X-Mailer: git-send-email 2.25.1
In-Reply-To: <20200331192945.2466880-1-jerinj@marvell.com>
References: <20200326165644.866053-1-jerinj@marvell.com>
<20200331192945.2466880-1-jerinj@marvell.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Content-Type: text/plain
X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.676
definitions=2020-03-31_07:2020-03-31,
2020-03-31 signatures=0
Subject: [dpdk-dev] [PATCH v3 20/29] node: ipv4 lookup for arm64
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
From: Nithin Dabilpuram <ndabilpuram@marvell.com>
Add arm64 specific IPv4 lookup process function
for ip4_lookup node. This node performs LPM lookup
on every packet received and forwards it to a next
node that is identified by lookup result.
Signed-off-by: Nithin Dabilpuram <ndabilpuram@marvell.com>
Signed-off-by: Kiran Kumar K <kirankumark@marvell.com>
Signed-off-by: Pavan Nikhilesh <pbhagavatula@marvell.com>
---
lib/librte_node/ip4_lookup.c | 6 +
lib/librte_node/ip4_lookup_neon.h | 238 ++++++++++++++++++++++++++++++
2 files changed, 244 insertions(+)
create mode 100644 lib/librte_node/ip4_lookup_neon.h
diff --git a/lib/librte_node/ip4_lookup.c b/lib/librte_node/ip4_lookup.c
index a91d42423..e8732e31b 100644
--- a/lib/librte_node/ip4_lookup.c
+++ b/lib/librte_node/ip4_lookup.c
@@ -28,6 +28,10 @@ struct ip4_lookup_node_main {
struct rte_lpm *lpm_tbl[RTE_MAX_NUMA_NODES];
};
+#if defined(RTE_MACHINE_CPUFLAG_NEON)
+#include "ip4_lookup_neon.h"
+#else
+
static uint16_t
ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
void **objs, uint16_t nb_objs)
@@ -101,6 +105,8 @@ ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
return nb_objs;
}
+#endif
+
static int
ip4_lookup_node_init(const struct rte_graph *graph, struct rte_node *node)
{
diff --git a/lib/librte_node/ip4_lookup_neon.h b/lib/librte_node/ip4_lookup_neon.h
new file mode 100644
index 000000000..0aafe4db2
--- /dev/null
+++ b/lib/librte_node/ip4_lookup_neon.h
@@ -0,0 +1,238 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2020 Marvell International Ltd.
+ */
+
+#ifndef __INCLUDE_IP4_LOOKUP_NEON_H__
+#define __INCLUDE_IP4_LOOKUP_NEON_H__
+
+/* ARM64 NEON */
+static uint16_t
+ip4_lookup_node_process(struct rte_graph *graph, struct rte_node *node,
+ void **objs, uint16_t nb_objs)
+{
+ struct rte_mbuf *mbuf0, *mbuf1, *mbuf2, *mbuf3, **pkts;
+ struct rte_ipv4_hdr *ipv4_hdr;
+ void **to_next, **from;
+ uint16_t last_spec = 0;
+ rte_edge_t next_index;
+ uint16_t n_left_from;
+ struct rte_lpm *lpm;
+ uint16_t held = 0;
+ uint32_t drop_nh;
+ rte_xmm_t result;
+ rte_xmm_t priv01;
+ rte_xmm_t priv23;
+ int32x4_t dip;
+ int rc, i;
+
+ /* Speculative next */
+ next_index = RTE_NODE_IP4_LOOKUP_NEXT_REWRITE;
+ /* Drop node */
+ drop_nh = ((uint32_t)RTE_NODE_IP4_LOOKUP_NEXT_PKT_DROP) << 16;
+
+ /* Get socket specific LPM from ctx */
+ lpm = *((struct rte_lpm **)node->ctx);
+
+ pkts = (struct rte_mbuf **)objs;
+ from = objs;
+ n_left_from = nb_objs;
+
+#define OBJS_PER_CLINE (RTE_CACHE_LINE_SIZE / sizeof(void *))
+ for (i = OBJS_PER_CLINE; i < RTE_GRAPH_BURST_SIZE; i += OBJS_PER_CLINE)
+ rte_prefetch0(&objs[i]);
+
+ for (i = 0; i < 4 && i < n_left_from; i++)
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[i], void *,
+ sizeof(struct rte_ether_hdr)));
+
+ /* Get stream for the speculated next node */
+ to_next = rte_node_next_stream_get(graph, node, next_index, nb_objs);
+ while (n_left_from >= 4) {
+#if RTE_GRAPH_BURST_SIZE > 64
+ /* Prefetch next-next mbufs */
+ if (likely(n_left_from > 11)) {
+ rte_prefetch0(pkts[8]);
+ rte_prefetch0(pkts[9]);
+ rte_prefetch0(pkts[10]);
+ rte_prefetch0(pkts[11]);
+ }
+#endif
+ /* Prefetch next mbuf data */
+ if (likely(n_left_from > 7)) {
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[4], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[5], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[6], void *,
+ sizeof(struct rte_ether_hdr)));
+ rte_prefetch0(rte_pktmbuf_mtod_offset(pkts[7], void *,
+ sizeof(struct rte_ether_hdr)));
+ }
+
+ mbuf0 = pkts[0];
+ mbuf1 = pkts[1];
+ mbuf2 = pkts[2];
+ mbuf3 = pkts[3];
+
+ pkts += 4;
+ n_left_from -= 4;
+
+ /* Extract DIP of mbuf0 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 0);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv01.u16[1] = ipv4_hdr->time_to_live;
+ priv01.u32[1] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf1 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf1, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 1);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv01.u16[5] = ipv4_hdr->time_to_live;
+ priv01.u32[3] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf2 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf2, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 2);
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv23.u16[1] = ipv4_hdr->time_to_live;
+ priv23.u32[1] = ipv4_hdr->hdr_checksum;
+
+ /* Extract DIP of mbuf3 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf3, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ dip = vsetq_lane_s32(ipv4_hdr->dst_addr, dip, 3);
+
+ dip = vreinterpretq_s32_u8(
+ vrev32q_u8(vreinterpretq_u8_s32(dip)));
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ priv23.u16[5] = ipv4_hdr->time_to_live;
+ priv23.u32[3] = ipv4_hdr->hdr_checksum;
+
+ /* Perform LPM lookup to get NH and next node */
+ rte_lpm_lookupx4(lpm, dip, result.u32, drop_nh);
+ priv01.u16[0] = result.u16[0];
+ priv01.u16[4] = result.u16[2];
+ priv23.u16[0] = result.u16[4];
+ priv23.u16[4] = result.u16[6];
+
+ rte_node_mbuf_priv1(mbuf0)->u = priv01.u64[0];
+ rte_node_mbuf_priv1(mbuf1)->u = priv01.u64[1];
+ rte_node_mbuf_priv1(mbuf2)->u = priv23.u64[0];
+ rte_node_mbuf_priv1(mbuf3)->u = priv23.u64[1];
+
+ /* Enqueue four to next node */
+ rte_edge_t fix_spec = ((next_index == result.u16[1]) &&
+ (result.u16[1] == result.u16[3]) &&
+ (result.u16[3] == result.u16[5]) &&
+ (result.u16[5] == result.u16[7]));
+
+ if (unlikely(fix_spec == 0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ /* Next0 */
+ if (next_index == result.u16[1]) {
+ to_next[0] = from[0];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[1],
+ from[0]);
+ }
+
+ /* Next1 */
+ if (next_index == result.u16[3]) {
+ to_next[0] = from[1];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[3],
+ from[1]);
+ }
+
+ /* Next2 */
+ if (next_index == result.u16[5]) {
+ to_next[0] = from[2];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[5],
+ from[2]);
+ }
+
+ /* Next3 */
+ if (next_index == result.u16[7]) {
+ to_next[0] = from[3];
+ to_next++;
+ held++;
+ } else {
+ rte_node_enqueue_x1(graph, node, result.u16[7],
+ from[3]);
+ }
+
+ from += 4;
+ } else {
+ last_spec += 4;
+ }
+ }
+
+ while (n_left_from > 0) {
+ uint32_t next_hop;
+ uint16_t next0;
+
+ mbuf0 = pkts[0];
+
+ pkts += 1;
+ n_left_from -= 1;
+
+ /* Extract DIP of mbuf0 */
+ ipv4_hdr = rte_pktmbuf_mtod_offset(mbuf0, struct rte_ipv4_hdr *,
+ sizeof(struct rte_ether_hdr));
+ /* Extract cksum, ttl as ipv4 hdr is in cache */
+ rte_node_mbuf_priv1(mbuf0)->cksum = ipv4_hdr->hdr_checksum;
+ rte_node_mbuf_priv1(mbuf0)->ttl = ipv4_hdr->time_to_live;
+
+ rc = rte_lpm_lookup(lpm, rte_be_to_cpu_32(ipv4_hdr->dst_addr),
+ &next_hop);
+ next_hop = (rc == 0) ? next_hop : drop_nh;
+
+ rte_node_mbuf_priv1(mbuf0)->nh = (uint16_t)next_hop;
+ next_hop = next_hop >> 16;
+ next0 = (uint16_t)next_hop;
+
+ if (unlikely(next_index ^ next0)) {
+ /* Copy things successfully speculated till now */
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ from += last_spec;
+ to_next += last_spec;
+ held += last_spec;
+ last_spec = 0;
+
+ rte_node_enqueue_x1(graph, node, next0, from[0]);
+ from += 1;
+ } else {
+ last_spec += 1;
+ }
+ }
+
+ /* !!! Home run !!! */
+ if (likely(last_spec == nb_objs)) {
+ rte_node_next_stream_move(graph, node, next_index);
+ return nb_objs;
+ }
+ held += last_spec;
+ rte_memcpy(to_next, from, last_spec * sizeof(from[0]));
+ rte_node_next_stream_put(graph, node, next_index, held);
+
+ return nb_objs;
+}
+
+#endif /* __INCLUDE_IP4_LOOKUP_NEON_H__ */
--
2.25.1