From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 24652A00C5; Wed, 13 May 2020 19:46:39 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id E87221D655; Wed, 13 May 2020 19:46:37 +0200 (CEST) Received: from mx0b-0016f401.pphosted.com (mx0b-0016f401.pphosted.com [67.231.156.173]) by dpdk.org (Postfix) with ESMTP id 181321D654 for ; Wed, 13 May 2020 19:46:36 +0200 (CEST) Received: from pps.filterd (m0045851.ppops.net [127.0.0.1]) by mx0b-0016f401.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 04DHQE1f012513; Wed, 13 May 2020 10:46:36 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=marvell.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding : content-type; s=pfpt0818; bh=/SwrSiZUuuQBneICZ1P60tjAND2ZuVhmtvq/KkCW98o=; b=sy23xM9pK5KN0jigp26u0yY7BGc8AZKrwgxHTQKiG8rCfN0a2FbtZtWHosDFUti66F8U LaZ2l+TEwYIMVuai7GqZAXgA5JluiQZkbc2hxKh9Cq8xjnHe9SJBalvqkZhxhVYe8HCd 3vsjwku9phaofy8+Uj5aTOpn9hmENED3HlCZrmfbeNBCBFag8c4HG00ksalUoKctF6rm zog+P/LokVQref7xbMBawDg39dbqWXLErMAPtbDZDwxcCij8H06FuxyXJXJxtWlrmHct XLwV1AJ2ZBooWeT8kTnDbAu8QIwrJ/CN+1HKS7AE7dcmrSQYoHMJab7EFPBPXHnfSuDa cw== Received: from sc-exch02.marvell.com ([199.233.58.182]) by mx0b-0016f401.pphosted.com with ESMTP id 3100xjwwa4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT); Wed, 13 May 2020 10:46:35 -0700 Received: from DC5-EXCH02.marvell.com (10.69.176.39) by SC-EXCH02.marvell.com (10.93.176.82) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 13 May 2020 10:46:33 -0700 Received: from DC5-EXCH01.marvell.com (10.69.176.38) by DC5-EXCH02.marvell.com (10.69.176.39) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Wed, 13 May 2020 10:46:32 -0700 Received: from maili.marvell.com (10.69.176.80) by DC5-EXCH01.marvell.com (10.69.176.38) with Microsoft SMTP Server id 15.0.1497.2 via Frontend Transport; Wed, 13 May 2020 10:46:32 -0700 Received: from ajoseph83.caveonetworks.com (ajoseph83.caveonetworks.com [10.29.45.60]) by maili.marvell.com (Postfix) with ESMTP id 40E883F703F; Wed, 13 May 2020 10:46:30 -0700 (PDT) From: Anoob Joseph To: Akhil Goyal , Radu Nicolau CC: Anoob Joseph , Narayana Prasad , Konstantin Ananyev , Date: Wed, 13 May 2020 23:15:19 +0530 Message-ID: <1589391919-20388-1-git-send-email-anoobj@marvell.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1588769253-10405-1-git-send-email-anoobj@marvell.com> References: <1588769253-10405-1-git-send-email-anoobj@marvell.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.216, 18.0.676 definitions=2020-05-13_08:2020-05-13, 2020-05-13 signatures=0 Subject: [dpdk-dev] [PATCH v4] examples/ipsec-secgw: add per core packet stats X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Adding per core packet handling stats to analyze traffic distribution when multiple cores are engaged. Since aggregating the packet stats across cores would affect performance, keeping the feature disabled using compile time flags. Signed-off-by: Anoob Joseph --- v4: * Moved print stats routine to control thread * Added stats for rx/tx pkts per call * Replaced free_pkt(m) with free_pkts(&m, 1) v3: * Added wrapper functions for updating rx, tx & dropped counts * Updated free_pkts() so that stats is updated internally * Introduced similar free_pkt() function which updates stats and frees the packet * Moved all inline functions and macros to ipsec-secgw.h * Made STATS_INTERVAL macro to control the interval of the stats update. STATS_INTERVAL = 0 would disable the feature. v2: * Added lookup failure cases to drop count examples/ipsec-secgw/ipsec-secgw.c | 114 ++++++++++++++++++++++++++++------- examples/ipsec-secgw/ipsec-secgw.h | 66 ++++++++++++++++++++ examples/ipsec-secgw/ipsec.c | 20 +++--- examples/ipsec-secgw/ipsec_process.c | 11 +--- 4 files changed, 170 insertions(+), 41 deletions(-) diff --git a/examples/ipsec-secgw/ipsec-secgw.c b/examples/ipsec-secgw/ipsec-secgw.c index f777ce2..d2b5b2c 100644 --- a/examples/ipsec-secgw/ipsec-secgw.c +++ b/examples/ipsec-secgw/ipsec-secgw.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "event_helper.h" #include "ipsec.h" @@ -287,6 +288,70 @@ adjust_ipv6_pktlen(struct rte_mbuf *m, const struct rte_ipv6_hdr *iph, } } +#if (STATS_INTERVAL > 0) + +/* Print out statistics on packet distribution */ +static void +print_stats_cb(__rte_unused void *param) +{ + uint64_t total_packets_dropped, total_packets_tx, total_packets_rx; + float burst_percent, rx_per_call, tx_per_call; + unsigned int coreid; + + total_packets_dropped = 0; + total_packets_tx = 0; + total_packets_rx = 0; + + const char clr[] = { 27, '[', '2', 'J', '\0' }; + const char topLeft[] = { 27, '[', '1', ';', '1', 'H', '\0' }; + + /* Clear screen and move to top left */ + printf("%s%s", clr, topLeft); + + printf("\nCore statistics ===================================="); + + for (coreid = 0; coreid < RTE_MAX_LCORE; coreid++) { + /* skip disabled cores */ + if (rte_lcore_is_enabled(coreid) == 0) + continue; + burst_percent = (float)(core_statistics[coreid].burst_rx * 100)/ + core_statistics[coreid].rx; + rx_per_call = (float)(core_statistics[coreid].rx)/ + core_statistics[coreid].rx_call; + tx_per_call = (float)(core_statistics[coreid].tx)/ + core_statistics[coreid].tx_call; + printf("\nStatistics for core %u ------------------------------" + "\nPackets received: %20"PRIu64 + "\nPackets sent: %24"PRIu64 + "\nPackets dropped: %21"PRIu64 + "\nBurst percent: %23.2f" + "\nPackets per Rx call: %17.2f" + "\nPackets per Tx call: %17.2f", + coreid, + core_statistics[coreid].rx, + core_statistics[coreid].tx, + core_statistics[coreid].dropped, + burst_percent, + rx_per_call, + tx_per_call); + + total_packets_dropped += core_statistics[coreid].dropped; + total_packets_tx += core_statistics[coreid].tx; + total_packets_rx += core_statistics[coreid].rx; + } + printf("\nAggregate statistics ===============================" + "\nTotal packets received: %14"PRIu64 + "\nTotal packets sent: %18"PRIu64 + "\nTotal packets dropped: %15"PRIu64, + total_packets_rx, + total_packets_tx, + total_packets_dropped); + printf("\n====================================================\n"); + + rte_eal_alarm_set(STATS_INTERVAL * US_PER_S, print_stats_cb, NULL); +} +#endif /* STATS_INTERVAL */ + static inline void prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t) { @@ -332,7 +397,7 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t) /* drop packet when IPv6 header exceeds first segment length */ if (unlikely(l3len > pkt->data_len)) { - rte_pktmbuf_free(pkt); + free_pkts(&pkt, 1); return; } @@ -349,7 +414,7 @@ prepare_one_packet(struct rte_mbuf *pkt, struct ipsec_traffic *t) /* Unknown/Unsupported type, drop the packet */ RTE_LOG(ERR, IPSEC, "Unsupported packet type 0x%x\n", rte_be_to_cpu_16(eth->ether_type)); - rte_pktmbuf_free(pkt); + free_pkts(&pkt, 1); return; } @@ -476,9 +541,12 @@ send_burst(struct lcore_conf *qconf, uint16_t n, uint16_t port) prepare_tx_burst(m_table, n, port, qconf); ret = rte_eth_tx_burst(port, queueid, m_table, n); + + core_stats_update_tx(ret); + if (unlikely(ret < n)) { do { - rte_pktmbuf_free(m_table[ret]); + free_pkts(&m_table[ret], 1); } while (++ret < n); } @@ -524,7 +592,7 @@ send_fragment_packet(struct lcore_conf *qconf, struct rte_mbuf *m, "error code: %d\n", __func__, m->pkt_len, rte_errno); - rte_pktmbuf_free(m); + free_pkts(&m, 1); return len; } @@ -549,7 +617,7 @@ send_single_packet(struct rte_mbuf *m, uint16_t port, uint8_t proto) } else if (frag_tbl_sz > 0) len = send_fragment_packet(qconf, m, port, proto); else - rte_pktmbuf_free(m); + free_pkts(&m, 1); /* enough pkts to be sent */ if (unlikely(len == MAX_PKT_BURST)) { @@ -583,19 +651,19 @@ inbound_sp_sa(struct sp_ctx *sp, struct sa_ctx *sa, struct traffic_type *ip, continue; } if (res == DISCARD) { - rte_pktmbuf_free(m); + free_pkts(&m, 1); continue; } /* Only check SPI match for processed IPSec packets */ if (i < lim && ((m->ol_flags & PKT_RX_SEC_OFFLOAD) == 0)) { - rte_pktmbuf_free(m); + free_pkts(&m, 1); continue; } sa_idx = res - 1; if (!inbound_sa_check(sa, m, sa_idx)) { - rte_pktmbuf_free(m); + free_pkts(&m, 1); continue; } ip->pkts[j++] = m; @@ -630,7 +698,7 @@ split46_traffic(struct ipsec_traffic *trf, struct rte_mbuf *mb[], uint32_t num) offsetof(struct ip6_hdr, ip6_nxt)); n6++; } else - rte_pktmbuf_free(m); + free_pkts(&m, 1); } trf->ip4.num = n4; @@ -682,7 +750,7 @@ outbound_sp(struct sp_ctx *sp, struct traffic_type *ip, m = ip->pkts[i]; sa_idx = ip->res[i] - 1; if (ip->res[i] == DISCARD) - rte_pktmbuf_free(m); + free_pkts(&m, 1); else if (ip->res[i] == BYPASS) ip->pkts[j++] = m; else { @@ -701,8 +769,7 @@ process_pkts_outbound(struct ipsec_ctx *ipsec_ctx, uint16_t idx, nb_pkts_out, i; /* Drop any IPsec traffic from protected ports */ - for (i = 0; i < traffic->ipsec.num; i++) - rte_pktmbuf_free(traffic->ipsec.pkts[i]); + free_pkts(traffic->ipsec.pkts, traffic->ipsec.num); traffic->ipsec.num = 0; @@ -742,14 +809,12 @@ process_pkts_inbound_nosp(struct ipsec_ctx *ipsec_ctx, uint32_t nb_pkts_in, i, idx; /* Drop any IPv4 traffic from unprotected ports */ - for (i = 0; i < traffic->ip4.num; i++) - rte_pktmbuf_free(traffic->ip4.pkts[i]); + free_pkts(traffic->ip4.pkts, traffic->ip4.num); traffic->ip4.num = 0; /* Drop any IPv6 traffic from unprotected ports */ - for (i = 0; i < traffic->ip6.num; i++) - rte_pktmbuf_free(traffic->ip6.pkts[i]); + free_pkts(traffic->ip6.pkts, traffic->ip6.num); traffic->ip6.num = 0; @@ -785,8 +850,7 @@ process_pkts_outbound_nosp(struct ipsec_ctx *ipsec_ctx, struct ip *ip; /* Drop any IPsec traffic from protected ports */ - for (i = 0; i < traffic->ipsec.num; i++) - rte_pktmbuf_free(traffic->ipsec.pkts[i]); + free_pkts(traffic->ipsec.pkts, traffic->ipsec.num); n = 0; @@ -900,7 +964,7 @@ route4_pkts(struct rt_ctx *rt_ctx, struct rte_mbuf *pkts[], uint8_t nb_pkts) } if ((pkt_hop & RTE_LPM_LOOKUP_SUCCESS) == 0) { - rte_pktmbuf_free(pkts[i]); + free_pkts(&pkts[i], 1); continue; } send_single_packet(pkts[i], pkt_hop & 0xff, IPPROTO_IP); @@ -952,7 +1016,7 @@ route6_pkts(struct rt_ctx *rt_ctx, struct rte_mbuf *pkts[], uint8_t nb_pkts) } if (pkt_hop == -1) { - rte_pktmbuf_free(pkts[i]); + free_pkts(&pkts[i], 1); continue; } send_single_packet(pkts[i], pkt_hop & 0xff, IPPROTO_IPV6); @@ -1168,8 +1232,10 @@ ipsec_poll_mode_worker(void) nb_rx = rte_eth_rx_burst(portid, queueid, pkts, MAX_PKT_BURST); - if (nb_rx > 0) + if (nb_rx > 0) { + core_stats_update_rx(nb_rx); process_pkts(qconf, pkts, nb_rx, portid); + } /* dequeue and process completed crypto-ops */ if (is_unprotected_port(portid)) @@ -2916,6 +2982,12 @@ main(int32_t argc, char **argv) check_all_ports_link_status(enabled_port_mask); +#if (STATS_INTERVAL > 0) + rte_eal_alarm_set(STATS_INTERVAL * US_PER_S, print_stats_cb, NULL); +#else + RTE_LOG(INFO, IPSEC, "Stats display disabled\n"); +#endif /* STATS_INTERVAL */ + /* launch per-lcore init on every lcore */ rte_eal_mp_remote_launch(ipsec_launch_one_lcore, eh_conf, CALL_MASTER); RTE_LCORE_FOREACH_SLAVE(lcore_id) { diff --git a/examples/ipsec-secgw/ipsec-secgw.h b/examples/ipsec-secgw/ipsec-secgw.h index 4b53cb5..572a930 100644 --- a/examples/ipsec-secgw/ipsec-secgw.h +++ b/examples/ipsec-secgw/ipsec-secgw.h @@ -6,6 +6,10 @@ #include +#ifndef STATS_INTERVAL +#define STATS_INTERVAL 0 +#endif + #define NB_SOCKETS 4 #define MAX_PKT_BURST 32 @@ -69,6 +73,19 @@ struct ethaddr_info { uint64_t src, dst; }; +#if (STATS_INTERVAL > 0) +struct ipsec_core_statistics { + uint64_t tx; + uint64_t rx; + uint64_t rx_call; + uint64_t tx_call; + uint64_t dropped; + uint64_t burst_rx; +} __rte_cache_aligned; + +struct ipsec_core_statistics core_statistics[RTE_MAX_LCORE]; +#endif /* STATS_INTERVAL */ + extern struct ethaddr_info ethaddr_tbl[RTE_MAX_ETHPORTS]; /* Port mask to identify the unprotected ports */ @@ -85,4 +102,53 @@ is_unprotected_port(uint16_t port_id) return unprotected_port_mask & (1 << port_id); } +static inline void +core_stats_update_rx(int n) +{ +#if (STATS_INTERVAL > 0) + int lcore_id = rte_lcore_id(); + core_statistics[lcore_id].rx += n; + core_statistics[lcore_id].rx_call++; + if (n == MAX_PKT_BURST) + core_statistics[lcore_id].burst_rx += n; +#else + RTE_SET_USED(n); +#endif /* STATS_INTERVAL */ +} + +static inline void +core_stats_update_tx(int n) +{ +#if (STATS_INTERVAL > 0) + int lcore_id = rte_lcore_id(); + core_statistics[lcore_id].tx += n; + core_statistics[lcore_id].tx_call++; +#else + RTE_SET_USED(n); +#endif /* STATS_INTERVAL */ +} + +static inline void +core_stats_update_drop(int n) +{ +#if (STATS_INTERVAL > 0) + int lcore_id = rte_lcore_id(); + core_statistics[lcore_id].dropped += n; +#else + RTE_SET_USED(n); +#endif /* STATS_INTERVAL */ +} + +/* helper routine to free bulk of packets */ +static inline void +free_pkts(struct rte_mbuf *mb[], uint32_t n) +{ + uint32_t i; + + for (i = 0; i != n; i++) + rte_pktmbuf_free(mb[i]); + + core_stats_update_drop(n); +} + #endif /* _IPSEC_SECGW_H_ */ diff --git a/examples/ipsec-secgw/ipsec.c b/examples/ipsec-secgw/ipsec.c index bf88d80..01faa7a 100644 --- a/examples/ipsec-secgw/ipsec.c +++ b/examples/ipsec-secgw/ipsec.c @@ -500,7 +500,7 @@ enqueue_cop_burst(struct cdev_qp *cqp) cqp->id, cqp->qp, ret, len); /* drop packets that we fail to enqueue */ for (i = ret; i < len; i++) - rte_pktmbuf_free(cqp->buf[i]->sym->m_src); + free_pkts(&cqp->buf[i]->sym->m_src, 1); } cqp->in_flight += ret; cqp->len = 0; @@ -528,7 +528,7 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx, for (i = 0; i < nb_pkts; i++) { if (unlikely(sas[i] == NULL)) { - rte_pktmbuf_free(pkts[i]); + free_pkts(&pkts[i], 1); continue; } @@ -549,7 +549,7 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx, if ((unlikely(ips->security.ses == NULL)) && create_lookaside_session(ipsec_ctx, sa, ips)) { - rte_pktmbuf_free(pkts[i]); + free_pkts(&pkts[i], 1); continue; } @@ -563,7 +563,7 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx, case RTE_SECURITY_ACTION_TYPE_CPU_CRYPTO: RTE_LOG(ERR, IPSEC, "CPU crypto is not supported by the" " legacy mode."); - rte_pktmbuf_free(pkts[i]); + free_pkts(&pkts[i], 1); continue; case RTE_SECURITY_ACTION_TYPE_NONE: @@ -575,7 +575,7 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx, if ((unlikely(ips->crypto.ses == NULL)) && create_lookaside_session(ipsec_ctx, sa, ips)) { - rte_pktmbuf_free(pkts[i]); + free_pkts(&pkts[i], 1); continue; } @@ -584,7 +584,7 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx, ret = xform_func(pkts[i], sa, &priv->cop); if (unlikely(ret)) { - rte_pktmbuf_free(pkts[i]); + free_pkts(&pkts[i], 1); continue; } break; @@ -608,7 +608,7 @@ ipsec_enqueue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx, ret = xform_func(pkts[i], sa, &priv->cop); if (unlikely(ret)) { - rte_pktmbuf_free(pkts[i]); + free_pkts(&pkts[i], 1); continue; } @@ -643,7 +643,7 @@ ipsec_inline_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx, sa = priv->sa; ret = xform_func(pkt, sa, &priv->cop); if (unlikely(ret)) { - rte_pktmbuf_free(pkt); + free_pkts(&pkt, 1); continue; } pkts[nb_pkts++] = pkt; @@ -690,13 +690,13 @@ ipsec_dequeue(ipsec_xform_fn xform_func, struct ipsec_ctx *ipsec_ctx, RTE_SECURITY_ACTION_TYPE_NONE) { ret = xform_func(pkt, sa, cops[j]); if (unlikely(ret)) { - rte_pktmbuf_free(pkt); + free_pkts(&pkt, 1); continue; } } else if (ipsec_get_action_type(sa) == RTE_SECURITY_ACTION_TYPE_LOOKASIDE_PROTOCOL) { if (cops[j]->status) { - rte_pktmbuf_free(pkt); + free_pkts(&pkt, 1); continue; } } diff --git a/examples/ipsec-secgw/ipsec_process.c b/examples/ipsec-secgw/ipsec_process.c index 6d3a3c9..5012e1a 100644 --- a/examples/ipsec-secgw/ipsec_process.c +++ b/examples/ipsec-secgw/ipsec_process.c @@ -12,22 +12,13 @@ #include #include "ipsec.h" +#include "ipsec-secgw.h" #define SATP_OUT_IPV4(t) \ ((((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TRANS && \ (((t) & RTE_IPSEC_SATP_IPV_MASK) == RTE_IPSEC_SATP_IPV4)) || \ ((t) & RTE_IPSEC_SATP_MODE_MASK) == RTE_IPSEC_SATP_MODE_TUNLV4) -/* helper routine to free bulk of packets */ -static inline void -free_pkts(struct rte_mbuf *mb[], uint32_t n) -{ - uint32_t i; - - for (i = 0; i != n; i++) - rte_pktmbuf_free(mb[i]); -} - /* helper routine to free bulk of crypto-ops and related packets */ static inline void free_cops(struct rte_crypto_op *cop[], uint32_t n) -- 2.7.4