From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id ADBBD42D12; Wed, 21 Jun 2023 10:38:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 3488F406BC; Wed, 21 Jun 2023 10:38:46 +0200 (CEST) Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by mails.dpdk.org (Postfix) with ESMTP id 5C1DF4068E for ; Wed, 21 Jun 2023 10:38:44 +0200 (CEST) Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-666fb8b1bc8so4122636b3a.1 for ; Wed, 21 Jun 2023 01:38:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687336723; x=1689928723; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DWBr3u1kZv5rZvSgxnQ/DuYtVRvzrZM82bOsa0Ah26o=; b=mKRxgAyGV2epQeyZX3B9scZ9ZjbBUSAZNWvyf5OhckpJDlqIf4EAqFEZtwC+cnW9Hv pCU+ZNReIVuOWlPEql7QrvTwBtppN4mytcoKl2S5EwOWlpG1/FS3SzeW7B9Deo4f4o29 EwUQvul6Kha+hbkbPL+woJ/+3YPlDTz0jRRryRGJHuboD7l4OLsTwI1eGIN5kqWS4TGI 9+KSUnsPm4UETaUpJ53OTPwHvpooH8sVXQzaqqBYv27doTxIl4YECP13NxrhKbM8NiwA /PbfK4Nm1LSpaa8n96IpQ1ZO9HnBdTTla8bmeZMnQnl8u0/TKusrNDcrCcF23pduBO95 qCpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687336723; x=1689928723; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DWBr3u1kZv5rZvSgxnQ/DuYtVRvzrZM82bOsa0Ah26o=; b=CY9BDnhVVqEr6ZAfIzVOX3Oa2n8rrnNhkBZV1vymrTuEb+Ha+nZmhWxNhvGT2PTNiH zobFNiLEn8jyWDWf8B5Nb6P3ouBHGX7uRDfmkPzyfUCW6xd/2ZEIVvEzk6TX89VkUUSD ThkXyghYD8SVFcyzIPP4mTNPBmA4hXjgDVr3R24XcKHVzSxvIIp8xBzsT+z1WWjAOCkW wxdfk12G7sRNGnHm+O6c1wU9u6Dj4R6oclMSrY8r0va9x6HIqe+jGqMGgHp86B8rUOFG VJMWf4drKYVeGt/flDNmGoaJb6b5Q4U3HBAJMfooK99dMORTGAmx6tXAHI81Zd++u/3p 0JqA== X-Gm-Message-State: AC+VfDzHqwsQhKEtwVdVQVKmkVH2TL7JyHIevwq6WtELvE0JtjSh0Pdn wbiB3W0zpsUzPuURUqESa9cTPGy9jkI= X-Google-Smtp-Source: ACHHUZ6or4ibXUBp72tBFhEt+/seC4bhXMDZbP61lypoQ/8ZhVie7s7KAsQf3Lm8n4jxJCYnw+40+A== X-Received: by 2002:a05:6a00:244a:b0:666:8cbb:6e02 with SMTP id d10-20020a056a00244a00b006668cbb6e02mr18516243pfj.8.1687336723147; Wed, 21 Jun 2023 01:38:43 -0700 (PDT) Received: from kparameshw7KFWX.vmware.com.com ([103.19.213.32]) by smtp.gmail.com with ESMTPSA id k10-20020aa792ca000000b00653dc27acadsm2436173pfa.205.2023.06.21.01.38.41 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 21 Jun 2023 01:38:42 -0700 (PDT) From: Kumara Parameshwaran To: jiayu.hu@intel.com Cc: dev@dpdk.org, Kumara Parameshwaran Subject: [PATCH v11 1/2] gro : refactor IPv4 to add GRO support for IPv6 Date: Wed, 21 Jun 2023 14:08:35 +0530 Message-Id: <20230621083836.63060-1-kumaraparamesh92@gmail.com> X-Mailer: git-send-email 2.37.1 (Apple Git-137.1) In-Reply-To: <20221020181425.48006-1-kumaraparmesh92@gmail.com> References: <20221020181425.48006-1-kumaraparmesh92@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Refactor the existing code to reuse data strutures and functions to add support for IPv6 GRO Signed-off-by: Kumara Parameshwaran --- v1: * Changes to support GRO for TCP/ipv6 packets. This does not include vxlan changes. * The GRO is performed only for ipv6 packets that does not contain extension headers. * The logic for the TCP coalescing remains the same, in ipv6 header the source address, destination address, flow label, version fields are expected to be the same. * Re-organised the code to reuse certain tcp functions for both ipv4 and ipv6 flows. v2: * Fix comments in gro_tcp6.h header file. v3: * Adderess review comments to fix code duplication for v4 and v6 v4: * Addresses review comments for v3, do not use callbacks v5: * Address review comments v6: * Fix warning and coding style issues v7: * Fix build compilation issue v8: * Use gro_tcp_internal.h for functions used for gro_tcp4 and gro_tcp6 and use gro_tcp.h for data structures and functions used across gro_vxlan_tcp4 v9: * Resolve merge conflict and add gro_tcp.h in proper path v10: * Refactor the code to contain 2 patches. This patch contains code/data strcuture changes in ipv4 code that can be reused. lib/gro/gro_tcp.h | 190 +++++++++++++++++++++++++++++++++++++ lib/gro/gro_tcp4.c | 177 ++++++++-------------------------- lib/gro/gro_tcp4.h | 170 ++------------------------------- lib/gro/gro_tcp_internal.h | 128 +++++++++++++++++++++++++ lib/gro/gro_vxlan_tcp4.c | 22 ++--- lib/gro/gro_vxlan_tcp4.h | 2 +- lib/gro/rte_gro.c | 2 +- 7 files changed, 373 insertions(+), 318 deletions(-) create mode 100644 lib/gro/gro_tcp.h create mode 100644 lib/gro/gro_tcp_internal.h diff --git a/lib/gro/gro_tcp.h b/lib/gro/gro_tcp.h new file mode 100644 index 0000000000..d926c4b8cc --- /dev/null +++ b/lib/gro/gro_tcp.h @@ -0,0 +1,190 @@ +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2023 Intel Corporation + */ +#ifndef _GRO_TCP_H_ +#define _GRO_TCP_H_ + +#define INVALID_ARRAY_INDEX 0xffffffffUL + +#include + +/* + * The max length of a IPv4 packet, which includes the length of the L3 + * header, the L4 header and the data payload. + */ +#define MAX_IP_PKT_LENGTH UINT16_MAX + +/* The maximum TCP header length */ +#define MAX_TCP_HLEN 60 +#define INVALID_TCP_HDRLEN(len) \ + (((len) < sizeof(struct rte_tcp_hdr)) || ((len) > MAX_TCP_HLEN)) + +struct cmn_tcp_key { + struct rte_ether_addr eth_saddr; + struct rte_ether_addr eth_daddr; + uint32_t recv_ack; + uint16_t src_port; + uint16_t dst_port; +}; + +#define ASSIGN_COMMON_TCP_KEY(k1, k2) \ + do {\ + rte_ether_addr_copy(&(k1->eth_saddr), &(k2->eth_saddr)); \ + rte_ether_addr_copy(&(k1->eth_daddr), &(k2->eth_daddr)); \ + k2->recv_ack = k1->recv_ack; \ + k2->src_port = k1->src_port; \ + k2->dst_port = k1->dst_port; \ + } while (0) + +struct gro_tcp_item { + /* + * The first MBUF segment of the packet. If the value + * is NULL, it means the item is empty. + */ + struct rte_mbuf *firstseg; + /* The last MBUF segment of the packet */ + struct rte_mbuf *lastseg; + /* + * The time when the first packet is inserted into the table. + * This value won't be updated, even if the packet is merged + * with other packets. + */ + uint64_t start_time; + /* + * next_pkt_idx is used to chain the packets that + * are in the same flow but can't be merged together + * (e.g. caused by packet reordering). + */ + uint32_t next_pkt_idx; + /* TCP sequence number of the packet */ + uint32_t sent_seq; + union { + /* IPv4 ID of the packet */ + uint16_t ip_id; + /* Unused field for IPv6 */ + uint16_t unused; + } l3; + /* the number of merged packets */ + uint16_t nb_merged; + /* Indicate if IPv4 ID can be ignored */ + uint8_t is_atomic; +}; + +/* + * Merge two TCP packets without updating checksums. + * If cmp is larger than 0, append the new packet to the + * original packet. Otherwise, pre-pend the new packet to + * the original packet. + */ +static inline int +merge_two_tcp_packets(struct gro_tcp_item *item, + struct rte_mbuf *pkt, + int cmp, + uint32_t sent_seq, + uint16_t ip_id, + uint16_t l2_offset) +{ + struct rte_mbuf *pkt_head, *pkt_tail, *lastseg; + uint16_t hdr_len, l2_len; + + if (cmp > 0) { + pkt_head = item->firstseg; + pkt_tail = pkt; + } else { + pkt_head = pkt; + pkt_tail = item->firstseg; + } + + /* check if the IPv4 packet length is greater than the max value */ + hdr_len = l2_offset + pkt_head->l2_len + pkt_head->l3_len + + pkt_head->l4_len; + l2_len = l2_offset > 0 ? pkt_head->outer_l2_len : pkt_head->l2_len; + if (unlikely(pkt_head->pkt_len - l2_len + pkt_tail->pkt_len - + hdr_len > MAX_IP_PKT_LENGTH)) + return 0; + + if (unlikely(pkt_head->nb_segs >= 20)) + return 0; + + /* remove the packet header for the tail packet */ + rte_pktmbuf_adj(pkt_tail, hdr_len); + + /* chain two packets together */ + if (cmp > 0) { + item->lastseg->next = pkt; + item->lastseg = rte_pktmbuf_lastseg(pkt); + /* update IP ID to the larger value */ + item->l3.ip_id = ip_id; + } else { + lastseg = rte_pktmbuf_lastseg(pkt); + lastseg->next = item->firstseg; + item->firstseg = pkt; + /* update sent_seq to the smaller value */ + item->sent_seq = sent_seq; + item->l3.ip_id = ip_id; + } + item->nb_merged++; + + /* update MBUF metadata for the merged packet */ + pkt_head->nb_segs += pkt_tail->nb_segs; + pkt_head->pkt_len += pkt_tail->pkt_len; + + return 1; +} + +/* + * Check if two TCP packets are neighbors. + */ +static inline int +check_seq_option(struct gro_tcp_item *item, + struct rte_tcp_hdr *tcph, + uint32_t sent_seq, + uint16_t ip_id, + uint16_t tcp_hl, + uint16_t tcp_dl, + uint16_t l2_offset, + uint8_t is_atomic) +{ + struct rte_mbuf *pkt_orig = item->firstseg; + char *iph_orig; + struct rte_tcp_hdr *tcph_orig; + uint16_t len, tcp_hl_orig; + + iph_orig = (char *)(rte_pktmbuf_mtod(pkt_orig, char *) + + l2_offset + pkt_orig->l2_len); + tcph_orig = (struct rte_tcp_hdr *)(iph_orig + pkt_orig->l3_len); + tcp_hl_orig = pkt_orig->l4_len; + + /* Check if TCP option fields equal */ + len = RTE_MAX(tcp_hl, tcp_hl_orig) - sizeof(struct rte_tcp_hdr); + if ((tcp_hl != tcp_hl_orig) || ((len > 0) && + (memcmp(tcph + 1, tcph_orig + 1, + len) != 0))) + return 0; + + /* Don't merge packets whose DF bits are different */ + if (unlikely(item->is_atomic ^ is_atomic)) + return 0; + + /* check if the two packets are neighbors */ + len = pkt_orig->pkt_len - l2_offset - pkt_orig->l2_len - + pkt_orig->l3_len - tcp_hl_orig; + if ((sent_seq == item->sent_seq + len) && (is_atomic || + (ip_id == item->l3.ip_id + 1))) + /* append the new packet */ + return 1; + else if ((sent_seq + tcp_dl == item->sent_seq) && (is_atomic || + (ip_id + item->nb_merged == item->l3.ip_id))) + /* pre-pend the new packet */ + return -1; + + return 0; +} + +static inline int +is_same_common_tcp_key(struct cmn_tcp_key *k1, struct cmn_tcp_key *k2) +{ + return (!memcmp(k1, k2, sizeof(struct cmn_tcp_key))); +} + +#endif diff --git a/lib/gro/gro_tcp4.c b/lib/gro/gro_tcp4.c index 0014096e63..6645de592b 100644 --- a/lib/gro/gro_tcp4.c +++ b/lib/gro/gro_tcp4.c @@ -7,6 +7,7 @@ #include #include "gro_tcp4.h" +#include "gro_tcp_internal.h" void * gro_tcp4_tbl_create(uint16_t socket_id, @@ -30,7 +31,7 @@ gro_tcp4_tbl_create(uint16_t socket_id, if (tbl == NULL) return NULL; - size = sizeof(struct gro_tcp4_item) * entries_num; + size = sizeof(struct gro_tcp_item) * entries_num; tbl->items = rte_zmalloc_socket(__func__, size, RTE_CACHE_LINE_SIZE, @@ -71,18 +72,6 @@ gro_tcp4_tbl_destroy(void *tbl) rte_free(tcp_tbl); } -static inline uint32_t -find_an_empty_item(struct gro_tcp4_tbl *tbl) -{ - uint32_t i; - uint32_t max_item_num = tbl->max_item_num; - - for (i = 0; i < max_item_num; i++) - if (tbl->items[i].firstseg == NULL) - return i; - return INVALID_ARRAY_INDEX; -} - static inline uint32_t find_an_empty_flow(struct gro_tcp4_tbl *tbl) { @@ -95,56 +84,6 @@ find_an_empty_flow(struct gro_tcp4_tbl *tbl) return INVALID_ARRAY_INDEX; } -static inline uint32_t -insert_new_item(struct gro_tcp4_tbl *tbl, - struct rte_mbuf *pkt, - uint64_t start_time, - uint32_t prev_idx, - uint32_t sent_seq, - uint16_t ip_id, - uint8_t is_atomic) -{ - uint32_t item_idx; - - item_idx = find_an_empty_item(tbl); - if (item_idx == INVALID_ARRAY_INDEX) - return INVALID_ARRAY_INDEX; - - tbl->items[item_idx].firstseg = pkt; - tbl->items[item_idx].lastseg = rte_pktmbuf_lastseg(pkt); - tbl->items[item_idx].start_time = start_time; - tbl->items[item_idx].next_pkt_idx = INVALID_ARRAY_INDEX; - tbl->items[item_idx].sent_seq = sent_seq; - tbl->items[item_idx].ip_id = ip_id; - tbl->items[item_idx].nb_merged = 1; - tbl->items[item_idx].is_atomic = is_atomic; - tbl->item_num++; - - /* if the previous packet exists, chain them together. */ - if (prev_idx != INVALID_ARRAY_INDEX) { - tbl->items[item_idx].next_pkt_idx = - tbl->items[prev_idx].next_pkt_idx; - tbl->items[prev_idx].next_pkt_idx = item_idx; - } - - return item_idx; -} - -static inline uint32_t -delete_item(struct gro_tcp4_tbl *tbl, uint32_t item_idx, - uint32_t prev_item_idx) -{ - uint32_t next_idx = tbl->items[item_idx].next_pkt_idx; - - /* NULL indicates an empty item */ - tbl->items[item_idx].firstseg = NULL; - tbl->item_num--; - if (prev_item_idx != INVALID_ARRAY_INDEX) - tbl->items[prev_item_idx].next_pkt_idx = next_idx; - - return next_idx; -} - static inline uint32_t insert_new_flow(struct gro_tcp4_tbl *tbl, struct tcp4_flow_key *src, @@ -159,13 +98,10 @@ insert_new_flow(struct gro_tcp4_tbl *tbl, dst = &(tbl->flows[flow_idx].key); - rte_ether_addr_copy(&(src->eth_saddr), &(dst->eth_saddr)); - rte_ether_addr_copy(&(src->eth_daddr), &(dst->eth_daddr)); + ASSIGN_COMMON_TCP_KEY((&src->cmn_key), (&dst->cmn_key)); + dst->ip_src_addr = src->ip_src_addr; dst->ip_dst_addr = src->ip_dst_addr; - dst->recv_ack = src->recv_ack; - dst->src_port = src->src_port; - dst->dst_port = src->dst_port; tbl->flows[flow_idx].start_index = item_idx; tbl->flow_num++; @@ -173,21 +109,6 @@ insert_new_flow(struct gro_tcp4_tbl *tbl, return flow_idx; } -/* - * update the packet length for the flushed packet. - */ -static inline void -update_header(struct gro_tcp4_item *item) -{ - struct rte_ipv4_hdr *ipv4_hdr; - struct rte_mbuf *pkt = item->firstseg; - - ipv4_hdr = (struct rte_ipv4_hdr *)(rte_pktmbuf_mtod(pkt, char *) + - pkt->l2_len); - ipv4_hdr->total_length = rte_cpu_to_be_16(pkt->pkt_len - - pkt->l2_len); -} - int32_t gro_tcp4_reassemble(struct rte_mbuf *pkt, struct gro_tcp4_tbl *tbl, @@ -202,9 +123,8 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, uint8_t is_atomic; struct tcp4_flow_key key; - uint32_t cur_idx, prev_idx, item_idx; + uint32_t item_idx; uint32_t i, max_flow_num, remaining_flow_num; - int cmp; uint8_t find; /* @@ -239,6 +159,14 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, if (tcp_dl <= 0) return -1; + rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.cmn_key.eth_saddr)); + rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.cmn_key.eth_daddr)); + key.ip_src_addr = ipv4_hdr->src_addr; + key.ip_dst_addr = ipv4_hdr->dst_addr; + key.cmn_key.src_port = tcp_hdr->src_port; + key.cmn_key.dst_port = tcp_hdr->dst_port; + key.cmn_key.recv_ack = tcp_hdr->recv_ack; + /* * Save IPv4 ID for the packet whose DF bit is 0. For the packet * whose DF bit is 1, IPv4 ID is ignored. @@ -246,15 +174,6 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, frag_off = rte_be_to_cpu_16(ipv4_hdr->fragment_offset); is_atomic = (frag_off & RTE_IPV4_HDR_DF_FLAG) == RTE_IPV4_HDR_DF_FLAG; ip_id = is_atomic ? 0 : rte_be_to_cpu_16(ipv4_hdr->packet_id); - sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); - - rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.eth_saddr)); - rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.eth_daddr)); - key.ip_src_addr = ipv4_hdr->src_addr; - key.ip_dst_addr = ipv4_hdr->dst_addr; - key.src_port = tcp_hdr->src_port; - key.dst_port = tcp_hdr->dst_port; - key.recv_ack = tcp_hdr->recv_ack; /* Search for a matched flow. */ max_flow_num = tbl->max_flow_num; @@ -270,63 +189,44 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, } } - /* - * Fail to find a matched flow. Insert a new flow and store the - * packet into the flow. - */ if (find == 0) { - item_idx = insert_new_item(tbl, pkt, start_time, - INVALID_ARRAY_INDEX, sent_seq, ip_id, - is_atomic); + sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); + item_idx = insert_new_tcp_item(pkt, tbl->items, &tbl->item_num, + tbl->max_item_num, start_time, + INVALID_ARRAY_INDEX, sent_seq, ip_id, + is_atomic); if (item_idx == INVALID_ARRAY_INDEX) return -1; if (insert_new_flow(tbl, &key, item_idx) == - INVALID_ARRAY_INDEX) { + INVALID_ARRAY_INDEX) { /* * Fail to insert a new flow, so delete the * stored packet. - */ - delete_item(tbl, item_idx, INVALID_ARRAY_INDEX); + */ + delete_tcp_item(tbl->items, item_idx, &tbl->item_num, INVALID_ARRAY_INDEX); return -1; } return 0; } - /* - * Check all packets in the flow and try to find a neighbor for - * the input packet. - */ - cur_idx = tbl->flows[i].start_index; - prev_idx = cur_idx; - do { - cmp = check_seq_option(&(tbl->items[cur_idx]), tcp_hdr, - sent_seq, ip_id, pkt->l4_len, tcp_dl, 0, - is_atomic); - if (cmp) { - if (merge_two_tcp4_packets(&(tbl->items[cur_idx]), - pkt, cmp, sent_seq, ip_id, 0)) - return 1; - /* - * Fail to merge the two packets, as the packet - * length is greater than the max value. Store - * the packet into the flow. - */ - if (insert_new_item(tbl, pkt, start_time, cur_idx, - sent_seq, ip_id, is_atomic) == - INVALID_ARRAY_INDEX) - return -1; - return 0; - } - prev_idx = cur_idx; - cur_idx = tbl->items[cur_idx].next_pkt_idx; - } while (cur_idx != INVALID_ARRAY_INDEX); + return process_tcp_item(pkt, tcp_hdr, tcp_dl, tbl->items, tbl->flows[i].start_index, + &tbl->item_num, tbl->max_item_num, + ip_id, is_atomic, start_time); +} - /* Fail to find a neighbor, so store the packet into the flow. */ - if (insert_new_item(tbl, pkt, start_time, prev_idx, sent_seq, - ip_id, is_atomic) == INVALID_ARRAY_INDEX) - return -1; +/* + * update the packet length for the flushed packet. + */ +static inline void +update_header(struct gro_tcp_item *item) +{ + struct rte_ipv4_hdr *ipv4_hdr; + struct rte_mbuf *pkt = item->firstseg; - return 0; + ipv4_hdr = (struct rte_ipv4_hdr *)(rte_pktmbuf_mtod(pkt, char *) + + pkt->l2_len); + ipv4_hdr->total_length = rte_cpu_to_be_16(pkt->pkt_len - + pkt->l2_len); } uint16_t @@ -353,7 +253,8 @@ gro_tcp4_tbl_timeout_flush(struct gro_tcp4_tbl *tbl, * Delete the packet and get the next * packet in the flow. */ - j = delete_item(tbl, j, INVALID_ARRAY_INDEX); + j = delete_tcp_item(tbl->items, j, + &tbl->item_num, INVALID_ARRAY_INDEX); tbl->flows[i].start_index = j; if (j == INVALID_ARRAY_INDEX) tbl->flow_num--; diff --git a/lib/gro/gro_tcp4.h b/lib/gro/gro_tcp4.h index 212f97a042..245e5da486 100644 --- a/lib/gro/gro_tcp4.h +++ b/lib/gro/gro_tcp4.h @@ -5,32 +5,15 @@ #ifndef _GRO_TCP4_H_ #define _GRO_TCP4_H_ -#include +#include "gro_tcp.h" -#define INVALID_ARRAY_INDEX 0xffffffffUL #define GRO_TCP4_TBL_MAX_ITEM_NUM (1024UL * 1024UL) -/* - * The max length of a IPv4 packet, which includes the length of the L3 - * header, the L4 header and the data payload. - */ -#define MAX_IPV4_PKT_LENGTH UINT16_MAX - -/* The maximum TCP header length */ -#define MAX_TCP_HLEN 60 -#define INVALID_TCP_HDRLEN(len) \ - (((len) < sizeof(struct rte_tcp_hdr)) || ((len) > MAX_TCP_HLEN)) - -/* Header fields representing a TCP/IPv4 flow */ +/* Header fields representing common fields in TCP flow */ struct tcp4_flow_key { - struct rte_ether_addr eth_saddr; - struct rte_ether_addr eth_daddr; + struct cmn_tcp_key cmn_key; uint32_t ip_src_addr; uint32_t ip_dst_addr; - - uint32_t recv_ack; - uint16_t src_port; - uint16_t dst_port; }; struct gro_tcp4_flow { @@ -42,42 +25,12 @@ struct gro_tcp4_flow { uint32_t start_index; }; -struct gro_tcp4_item { - /* - * The first MBUF segment of the packet. If the value - * is NULL, it means the item is empty. - */ - struct rte_mbuf *firstseg; - /* The last MBUF segment of the packet */ - struct rte_mbuf *lastseg; - /* - * The time when the first packet is inserted into the table. - * This value won't be updated, even if the packet is merged - * with other packets. - */ - uint64_t start_time; - /* - * next_pkt_idx is used to chain the packets that - * are in the same flow but can't be merged together - * (e.g. caused by packet reordering). - */ - uint32_t next_pkt_idx; - /* TCP sequence number of the packet */ - uint32_t sent_seq; - /* IPv4 ID of the packet */ - uint16_t ip_id; - /* the number of merged packets */ - uint16_t nb_merged; - /* Indicate if IPv4 ID can be ignored */ - uint8_t is_atomic; -}; - /* * TCP/IPv4 reassembly table structure. */ struct gro_tcp4_tbl { /* item array */ - struct gro_tcp4_item *items; + struct gro_tcp_item *items; /* flow array */ struct gro_tcp4_flow *flows; /* current item number */ @@ -186,120 +139,9 @@ uint32_t gro_tcp4_tbl_pkt_count(void *tbl); static inline int is_same_tcp4_flow(struct tcp4_flow_key k1, struct tcp4_flow_key k2) { - return (rte_is_same_ether_addr(&k1.eth_saddr, &k2.eth_saddr) && - rte_is_same_ether_addr(&k1.eth_daddr, &k2.eth_daddr) && - (k1.ip_src_addr == k2.ip_src_addr) && + return ((k1.ip_src_addr == k2.ip_src_addr) && (k1.ip_dst_addr == k2.ip_dst_addr) && - (k1.recv_ack == k2.recv_ack) && - (k1.src_port == k2.src_port) && - (k1.dst_port == k2.dst_port)); + is_same_common_tcp_key(&k1.cmn_key, &k2.cmn_key)); } -/* - * Merge two TCP/IPv4 packets without updating checksums. - * If cmp is larger than 0, append the new packet to the - * original packet. Otherwise, pre-pend the new packet to - * the original packet. - */ -static inline int -merge_two_tcp4_packets(struct gro_tcp4_item *item, - struct rte_mbuf *pkt, - int cmp, - uint32_t sent_seq, - uint16_t ip_id, - uint16_t l2_offset) -{ - struct rte_mbuf *pkt_head, *pkt_tail, *lastseg; - uint16_t hdr_len, l2_len; - - if (cmp > 0) { - pkt_head = item->firstseg; - pkt_tail = pkt; - } else { - pkt_head = pkt; - pkt_tail = item->firstseg; - } - - /* check if the IPv4 packet length is greater than the max value */ - hdr_len = l2_offset + pkt_head->l2_len + pkt_head->l3_len + - pkt_head->l4_len; - l2_len = l2_offset > 0 ? pkt_head->outer_l2_len : pkt_head->l2_len; - if (unlikely(pkt_head->pkt_len - l2_len + pkt_tail->pkt_len - - hdr_len > MAX_IPV4_PKT_LENGTH)) - return 0; - - /* remove the packet header for the tail packet */ - rte_pktmbuf_adj(pkt_tail, hdr_len); - - /* chain two packets together */ - if (cmp > 0) { - item->lastseg->next = pkt; - item->lastseg = rte_pktmbuf_lastseg(pkt); - /* update IP ID to the larger value */ - item->ip_id = ip_id; - } else { - lastseg = rte_pktmbuf_lastseg(pkt); - lastseg->next = item->firstseg; - item->firstseg = pkt; - /* update sent_seq to the smaller value */ - item->sent_seq = sent_seq; - item->ip_id = ip_id; - } - item->nb_merged++; - - /* update MBUF metadata for the merged packet */ - pkt_head->nb_segs += pkt_tail->nb_segs; - pkt_head->pkt_len += pkt_tail->pkt_len; - - return 1; -} - -/* - * Check if two TCP/IPv4 packets are neighbors. - */ -static inline int -check_seq_option(struct gro_tcp4_item *item, - struct rte_tcp_hdr *tcph, - uint32_t sent_seq, - uint16_t ip_id, - uint16_t tcp_hl, - uint16_t tcp_dl, - uint16_t l2_offset, - uint8_t is_atomic) -{ - struct rte_mbuf *pkt_orig = item->firstseg; - struct rte_ipv4_hdr *iph_orig; - struct rte_tcp_hdr *tcph_orig; - uint16_t len, tcp_hl_orig; - - iph_orig = (struct rte_ipv4_hdr *)(rte_pktmbuf_mtod(pkt_orig, char *) + - l2_offset + pkt_orig->l2_len); - tcph_orig = (struct rte_tcp_hdr *)((char *)iph_orig + pkt_orig->l3_len); - tcp_hl_orig = pkt_orig->l4_len; - - /* Check if TCP option fields equal */ - len = RTE_MAX(tcp_hl, tcp_hl_orig) - sizeof(struct rte_tcp_hdr); - if ((tcp_hl != tcp_hl_orig) || ((len > 0) && - (memcmp(tcph + 1, tcph_orig + 1, - len) != 0))) - return 0; - - /* Don't merge packets whose DF bits are different */ - if (unlikely(item->is_atomic ^ is_atomic)) - return 0; - - /* check if the two packets are neighbors */ - len = pkt_orig->pkt_len - l2_offset - pkt_orig->l2_len - - pkt_orig->l3_len - tcp_hl_orig; - if ((sent_seq == item->sent_seq + len) && (is_atomic || - (ip_id == item->ip_id + 1))) - /* append the new packet */ - return 1; - else if ((sent_seq + tcp_dl == item->sent_seq) && (is_atomic || - (ip_id + item->nb_merged == item->ip_id))) - /* pre-pend the new packet */ - return -1; - - return 0; -} #endif diff --git a/lib/gro/gro_tcp_internal.h b/lib/gro/gro_tcp_internal.h new file mode 100644 index 0000000000..cc84abeaeb --- /dev/null +++ b/lib/gro/gro_tcp_internal.h @@ -0,0 +1,128 @@ + +/* SPDX-License-Identifier: BSD-3-Clause + * Copyright(c) 2023 Intel Corporation + */ + +#ifndef _GRO_TCP_INTERNAL_H_ +#define _GRO_TCP_INTERNAL_H_ + +static inline uint32_t +find_an_empty_item(struct gro_tcp_item *items, + uint32_t max_item_num) +{ + uint32_t i; + + for (i = 0; i < max_item_num; i++) + if (items[i].firstseg == NULL) + return i; + return INVALID_ARRAY_INDEX; +} + +static inline uint32_t +insert_new_tcp_item(struct rte_mbuf *pkt, + struct gro_tcp_item *items, + uint32_t *item_num, + uint32_t max_item_num, + uint64_t start_time, + uint32_t prev_idx, + uint32_t sent_seq, + uint16_t ip_id, + uint8_t is_atomic) +{ + uint32_t item_idx; + + item_idx = find_an_empty_item(items, max_item_num); + if (item_idx == INVALID_ARRAY_INDEX) + return INVALID_ARRAY_INDEX; + + items[item_idx].firstseg = pkt; + items[item_idx].lastseg = rte_pktmbuf_lastseg(pkt); + items[item_idx].start_time = start_time; + items[item_idx].next_pkt_idx = INVALID_ARRAY_INDEX; + items[item_idx].sent_seq = sent_seq; + items[item_idx].l3.ip_id = ip_id; + items[item_idx].nb_merged = 1; + items[item_idx].is_atomic = is_atomic; + (*item_num) += 1; + + /* if the previous packet exists, chain them together. */ + if (prev_idx != INVALID_ARRAY_INDEX) { + items[item_idx].next_pkt_idx = + items[prev_idx].next_pkt_idx; + items[prev_idx].next_pkt_idx = item_idx; + } + + return item_idx; +} + +static inline uint32_t +delete_tcp_item(struct gro_tcp_item *items, uint32_t item_idx, + uint32_t *item_num, + uint32_t prev_item_idx) +{ + uint32_t next_idx = items[item_idx].next_pkt_idx; + + /* NULL indicates an empty item */ + items[item_idx].firstseg = NULL; + (*item_num) -= 1; + if (prev_item_idx != INVALID_ARRAY_INDEX) + items[prev_item_idx].next_pkt_idx = next_idx; + + return next_idx; +} + +static inline int32_t +process_tcp_item(struct rte_mbuf *pkt, + struct rte_tcp_hdr *tcp_hdr, + int32_t tcp_dl, + struct gro_tcp_item *items, + uint32_t item_idx, + uint32_t *item_num, + uint32_t max_item_num, + uint16_t ip_id, + uint8_t is_atomic, + uint64_t start_time) +{ + uint32_t cur_idx; + uint32_t prev_idx; + int cmp; + uint32_t sent_seq; + + sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); + /* + * Check all packets in the flow and try to find a neighbor for + * the input packet. + */ + cur_idx = item_idx; + prev_idx = cur_idx; + do { + cmp = check_seq_option(&items[cur_idx], tcp_hdr, + sent_seq, ip_id, pkt->l4_len, tcp_dl, 0, + is_atomic); + if (cmp) { + if (merge_two_tcp_packets(&items[cur_idx], + pkt, cmp, sent_seq, ip_id, 0)) + return 1; + /* + * Fail to merge the two packets, as the packet + * length is greater than the max value. Store + * the packet into the flow. + */ + if (insert_new_tcp_item(pkt, items, item_num, max_item_num, + start_time, cur_idx, sent_seq, ip_id, is_atomic) == + INVALID_ARRAY_INDEX) + return -1; + return 0; + } + prev_idx = cur_idx; + cur_idx = items[cur_idx].next_pkt_idx; + } while (cur_idx != INVALID_ARRAY_INDEX); + + /* Fail to find a neighbor, so store the packet into the flow. */ + if (insert_new_tcp_item(pkt, items, item_num, max_item_num, start_time, prev_idx, sent_seq, + ip_id, is_atomic) == INVALID_ARRAY_INDEX) + return -1; + + return 0; +} +#endif diff --git a/lib/gro/gro_vxlan_tcp4.c b/lib/gro/gro_vxlan_tcp4.c index 3be4deb7c7..6ab7001922 100644 --- a/lib/gro/gro_vxlan_tcp4.c +++ b/lib/gro/gro_vxlan_tcp4.c @@ -116,7 +116,7 @@ insert_new_item(struct gro_vxlan_tcp4_tbl *tbl, tbl->items[item_idx].inner_item.start_time = start_time; tbl->items[item_idx].inner_item.next_pkt_idx = INVALID_ARRAY_INDEX; tbl->items[item_idx].inner_item.sent_seq = sent_seq; - tbl->items[item_idx].inner_item.ip_id = ip_id; + tbl->items[item_idx].inner_item.l3.ip_id = ip_id; tbl->items[item_idx].inner_item.nb_merged = 1; tbl->items[item_idx].inner_item.is_atomic = is_atomic; tbl->items[item_idx].outer_ip_id = outer_ip_id; @@ -163,15 +163,9 @@ insert_new_flow(struct gro_vxlan_tcp4_tbl *tbl, dst = &(tbl->flows[flow_idx].key); - rte_ether_addr_copy(&(src->inner_key.eth_saddr), - &(dst->inner_key.eth_saddr)); - rte_ether_addr_copy(&(src->inner_key.eth_daddr), - &(dst->inner_key.eth_daddr)); + ASSIGN_COMMON_TCP_KEY((&(src->inner_key.cmn_key)), (&(dst->inner_key.cmn_key))); dst->inner_key.ip_src_addr = src->inner_key.ip_src_addr; dst->inner_key.ip_dst_addr = src->inner_key.ip_dst_addr; - dst->inner_key.recv_ack = src->inner_key.recv_ack; - dst->inner_key.src_port = src->inner_key.src_port; - dst->inner_key.dst_port = src->inner_key.dst_port; dst->vxlan_hdr.vx_flags = src->vxlan_hdr.vx_flags; dst->vxlan_hdr.vx_vni = src->vxlan_hdr.vx_vni; @@ -248,7 +242,7 @@ merge_two_vxlan_tcp4_packets(struct gro_vxlan_tcp4_item *item, uint16_t outer_ip_id, uint16_t ip_id) { - if (merge_two_tcp4_packets(&item->inner_item, pkt, cmp, sent_seq, + if (merge_two_tcp_packets(&item->inner_item, pkt, cmp, sent_seq, ip_id, pkt->outer_l2_len + pkt->outer_l3_len)) { /* Update the outer IPv4 ID to the large value. */ @@ -357,13 +351,13 @@ gro_vxlan_tcp4_reassemble(struct rte_mbuf *pkt, sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); - rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.inner_key.eth_saddr)); - rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.inner_key.eth_daddr)); + rte_ether_addr_copy(&(eth_hdr->src_addr), &(key.inner_key.cmn_key.eth_saddr)); + rte_ether_addr_copy(&(eth_hdr->dst_addr), &(key.inner_key.cmn_key.eth_daddr)); key.inner_key.ip_src_addr = ipv4_hdr->src_addr; key.inner_key.ip_dst_addr = ipv4_hdr->dst_addr; - key.inner_key.recv_ack = tcp_hdr->recv_ack; - key.inner_key.src_port = tcp_hdr->src_port; - key.inner_key.dst_port = tcp_hdr->dst_port; + key.inner_key.cmn_key.recv_ack = tcp_hdr->recv_ack; + key.inner_key.cmn_key.src_port = tcp_hdr->src_port; + key.inner_key.cmn_key.dst_port = tcp_hdr->dst_port; key.vxlan_hdr.vx_flags = vxlan_hdr->vx_flags; key.vxlan_hdr.vx_vni = vxlan_hdr->vx_vni; diff --git a/lib/gro/gro_vxlan_tcp4.h b/lib/gro/gro_vxlan_tcp4.h index 7832942a68..662db01a88 100644 --- a/lib/gro/gro_vxlan_tcp4.h +++ b/lib/gro/gro_vxlan_tcp4.h @@ -36,7 +36,7 @@ struct gro_vxlan_tcp4_flow { }; struct gro_vxlan_tcp4_item { - struct gro_tcp4_item inner_item; + struct gro_tcp_item inner_item; /* IPv4 ID in the outer IPv4 header */ uint16_t outer_ip_id; /* Indicate if outer IPv4 ID can be ignored */ diff --git a/lib/gro/rte_gro.c b/lib/gro/rte_gro.c index e35399fd42..ac3d1cdc94 100644 --- a/lib/gro/rte_gro.c +++ b/lib/gro/rte_gro.c @@ -147,7 +147,7 @@ rte_gro_reassemble_burst(struct rte_mbuf **pkts, /* allocate a reassembly table for TCP/IPv4 GRO */ struct gro_tcp4_tbl tcp_tbl; struct gro_tcp4_flow tcp_flows[RTE_GRO_MAX_BURST_ITEM_NUM]; - struct gro_tcp4_item tcp_items[RTE_GRO_MAX_BURST_ITEM_NUM] = {{0} }; + struct gro_tcp_item tcp_items[RTE_GRO_MAX_BURST_ITEM_NUM] = {{0} }; /* allocate a reassembly table for UDP/IPv4 GRO */ struct gro_udp4_tbl udp_tbl; -- 2.25.1