From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 8D49942D04; Tue, 20 Jun 2023 09:36:24 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1A9F64113F; Tue, 20 Jun 2023 09:36:24 +0200 (CEST) Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mails.dpdk.org (Postfix) with ESMTP id 8A56F4068E for ; Tue, 20 Jun 2023 09:36:21 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1687246581; x=1718782581; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=w4Up1ungwtZnOu0RN9qjC7QRmbG4PE08LBiShfPHbTA=; b=DVdO01QO3+2rfiSKR4JtY9EkqUHeYXcbPpQHzjjf27YhnHTUnP1t+3vG eyRH5KYnJWfAFMjnXelXA6VcioZOW6pZPsybaffQUB1nNSIXlEOaDT8fP ttFMjmAsx24vQ6JxF2BohdTBq4CompLONjboDRG0+bcYGTKOFo0aKXksr /+2SNg7vJVmYv+Ob1Hg+CqmIdYytZtcF1DIg56HuT5+sbFMM/cCOf4wRp oH4J+5wpxbL1ZfTK5WHxwFamWccRHA5PweDGG5SKcMIhZ4CkPcklyVJxp ubzHKsk7zSfig6luV5ZESBlyBRBJK+2jnCnv7HiHb4bghf31kjRrTMs4J Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10746"; a="344534918" X-IronPort-AV: E=Sophos;i="6.00,256,1681196400"; d="scan'208";a="344534918" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 20 Jun 2023 00:35:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10746"; a="708156329" X-IronPort-AV: E=Sophos;i="6.00,256,1681196400"; d="scan'208";a="708156329" Received: from orsmsx603.amr.corp.intel.com ([10.22.229.16]) by orsmga007.jf.intel.com with ESMTP; 20 Jun 2023 00:35:59 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX603.amr.corp.intel.com (10.22.229.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Tue, 20 Jun 2023 00:35:59 -0700 Received: from orsmsx611.amr.corp.intel.com (10.22.229.24) by ORSMSX611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23; Tue, 20 Jun 2023 00:35:57 -0700 Received: from orsedg603.ED.cps.intel.com (10.7.248.4) by orsmsx611.amr.corp.intel.com (10.22.229.24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.23 via Frontend Transport; Tue, 20 Jun 2023 00:35:57 -0700 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (104.47.57.168) by edgegateway.intel.com (134.134.137.100) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.1.2507.23; Tue, 20 Jun 2023 00:35:57 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Roy+BdLHZC3/jVT0NNHklxBFziefW1XbimHZmQS59YxwUcqYpez0GUV482D6Zwoupdy5C5zTzApDYcfhEBJFUKl1ea4+DR81Ud9JRAw5/lgEAL/RPMP/6YyP/vNpPrfqZYpal7pf2eM1leBFSxOh1/9LbYs8Kg8fHfkSAmluttffXmU29CG9GGJ/bskr7OMQPhVud6QrGtgMsEg3tH6VOX0zMVc2iadWX76Qdfqyt0wIhbDknVDtWpN1x5+gV0MyPgQQSgkP1xqbP/jcmf8aUEJlbH5c69inRMHXhEcCKtgGFKZwfnK1y8myUncalsj6JnGlwYC3C7uDYzo6xFRqLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=op5gnQa1QTsem7DNxH/xUCOwwlt7LRNAk+bseOeVdEs=; b=WGL7fD6aphEOLbJQ8k8vCjHEnyuvESazMDSF40MNjcn9NdrWzaVAo1vQLC2a7wlluteGOoBxP+FffgpYy3QzB5WkIHjSnWdPrjUshbRLQyn6x1dUSIT7FmI8gayI8+WNtD40RI4gLMraNz9KTzDdiu+7WiM58V7Q+RaEdzDOATeO4RVtYmrfwXchcVlovA0GCSQqMEoQBPkKsvVZPvpXEEY/J5hCSSE3wVh+sIesednTqCzrYzKdAwBzmgXmfw1QfUKLT81y7ACA72jF1rS1az9GR2fw+nFj3Kndh0YLB0KqPeeA03bfZdkzVvUbQEO4PbDJDSuo12CAoFuZzA6jxA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none Received: from DS0PR11MB6494.namprd11.prod.outlook.com (2603:10b6:8:c2::14) by IA0PR11MB8334.namprd11.prod.outlook.com (2603:10b6:208:483::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6500.29; Tue, 20 Jun 2023 07:35:50 +0000 Received: from DS0PR11MB6494.namprd11.prod.outlook.com ([fe80::bbd0:7628:1b7c:78b0]) by DS0PR11MB6494.namprd11.prod.outlook.com ([fe80::bbd0:7628:1b7c:78b0%4]) with mapi id 15.20.6500.036; Tue, 20 Jun 2023 07:35:49 +0000 From: "Hu, Jiayu" To: Kumara Parameshwaran CC: "dev@dpdk.org" , Kumara Parameshwaran , "thomas@monjalon.net" Subject: RE: [PATCH v5] gro : fix reordering of packets in GRO library Thread-Topic: [PATCH v5] gro : fix reordering of packets in GRO library Thread-Index: AQHY7cBuFq3EALEykUaAHR2RKMn3DK+Uod4Q Date: Tue, 20 Jun 2023 07:35:49 +0000 Message-ID: References: <20221028095114.17210-1-kumaraparmesh92@gmail.com> <20221101070557.58808-1-kumaraparmesh92@gmail.com> In-Reply-To: <20221101070557.58808-1-kumaraparmesh92@gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=intel.com; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DS0PR11MB6494:EE_|IA0PR11MB8334:EE_ x-ms-office365-filtering-correlation-id: 12f18147-7058-4860-44d7-08db7160f7c9 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: oUgKPHRg34bLl7Be6E4HtUfwntQmhEvfRmsYok/6M8YpKTEzii3wXJl+pPShiyX0cQoerEC1fbJDe7CiG417ipfg5V5o9WMgldSwtMoNz1d/YQzG42EYU129z/oE41AH+cEMGlaF7QUAQwnLFfW5UtYZSTgHOZ61Yc3V+4+1MnnUQ2pNGOdIRVAhzrWLtUIzkvA+RwzFa++Lf8Wi+ZS4EFDHxad67C9CztojieuOn8a4gJXoDdYxt65N5A1Jx5Dq1tFjKdk8MMoAds9VwXeoLa7wpjwDhWDqTHJnC+A1iwbdGcAeQwFlvCBF7FVCqbyDsazeqTISbeIAjoxbKlF71vE+zN0NZAhS1ouQs6FpnwsisNvtVaCfFpG5pNiYfZixoGmbzuKu/Kwe0icCSg1MG6Xdkdm1XvYGP4PYMJnjNIgYse3qdMX+wburLFNv69ihFZ49gWV61XdMptxzaSJUTHvdKboxj/Mjeqv4/y5J+4p187i5BQrpCSbY1e1WI+1jJXpSyfvjVliUINtihbB152hYvRdrify0nhUkLV4VB8KxTU4le1lk8rXuGD6tXPOO6D+BeWq8YmdK/evuJQ+jRYtkHMSFRRXlPjYoGOadPrm6ba6YmcrHgvvB2Lhxi3v2 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:DS0PR11MB6494.namprd11.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230028)(396003)(136003)(366004)(346002)(39860400002)(376002)(451199021)(2906002)(64756008)(41300700001)(8676002)(55016003)(52536014)(66946007)(316002)(4326008)(5660300002)(8936002)(6916009)(66446008)(66556008)(76116006)(66476007)(33656002)(186003)(83380400001)(71200400001)(9686003)(6506007)(53546011)(26005)(38070700005)(82960400001)(122000001)(86362001)(478600001)(38100700002)(54906003)(7696005); DIR:OUT; SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?iBLFUSQW6n2pTL9txoy+B5egEQRrLpfH/tjpTrzhgcS+a1cpJ9fTvxvoTUOk?= =?us-ascii?Q?04umlRPXxl8wE0Zhw9oq1x2qLrQnlM0zuPL9JqPEoYPi0AXUI26oCPBLkeLx?= =?us-ascii?Q?Fx4CYbs0JRGhP8eectNTKnxi/7mAa5RXHA0q/j52e60wl0UWHfkU5IPVkef7?= =?us-ascii?Q?sTG4kcTdfbC8uWF+V0eM4YRdTslUJMZX6yb0hpK+6D35YanupsoQ7CFm6yzQ?= =?us-ascii?Q?rzoMQ2N/BpHPOvTVbctFZQLvwPGLmbG4W07AKZrdJUZMsEsObnBFE1Tgs7Sz?= =?us-ascii?Q?9rT82NPvaEDqOXYXoMLn5jGp+lSOnO88PIOPJIJiiBObd+84IWF3Z5hnOSbj?= =?us-ascii?Q?mYWePQs7jcHyerP15gHgj91MbYuZ/K9ajx5KmGSga+uKutMgVl0UVpbzoZFr?= =?us-ascii?Q?guMQfpigZzSKM1pDXxwQoTtduchSW/E7dJaKa0q5w8cLUH9tQVXoiFmhabkC?= =?us-ascii?Q?IbMhFkZNlAhVHToCdellfqa3oRyJDe46E/W4ZqEDB6JZ525x30IL9hyqJdA5?= =?us-ascii?Q?dqVb0weq/2/yD53yRWCyUncv1jHFwhZHF9s4C/hfntCVohAuHUEOMxulje5i?= =?us-ascii?Q?u0Oq39ETw63stQgiUQwzJkI0qBg7HRldYd98UsIbWRxlCPsEGT7GVj4glAjl?= =?us-ascii?Q?9fuEIn20SHiXOYmhQUtJluZ/3xckbYWTnjagR1P+hd7wZcxox202FVkl/rYa?= =?us-ascii?Q?EryTZTpSmMEdb6MMQR02yFJWPBECAoU+JR7XuHf6gNfIjBNhDyG9Y8loWquI?= =?us-ascii?Q?q5PuWSHKqwI047UrHfL25UcID2Xb57RQnglqqdpedjC57r86xWG5tw6AxBIH?= =?us-ascii?Q?95cUDgg7qjW6oGhqckNvl9J1hmtTLORm5dANyZ12nCFF3kUdooGkXXNf4f1g?= =?us-ascii?Q?Quj91yJSD5eikn5ckHPqzw4l7lVrHdAbQ5JvHnKTbsLgh6kLBVwa9+tcbTfq?= =?us-ascii?Q?/QOtKM8xbpFmj1uhd51ZvNt7Y9wMtL6a0AJBNu0yZSVrG75p5+DPvyuTuAR/?= =?us-ascii?Q?0ClQJ6/wBpcedr2Fg20JGRYbXAx8VVSWZQquQZ9bXx8QBuxJqAZvcZWIoMnp?= =?us-ascii?Q?mJkAAOKZ81WAGBJ37n6qOSf5LdzzaHnUq5WI0ulG+5oMCKZggdrJfdqmBmJe?= =?us-ascii?Q?SAr2Rfhr+uioAnj695B2L4bg8YcpkKJt+LyM6bFXXZb3DZOIKSwnrmMG3CKX?= =?us-ascii?Q?wwREzZ0D0CJxOba1Fn0ieDE/3FxQJj/H7/s1HH2tTBwmV+gv9rZQFTzF2ibl?= =?us-ascii?Q?7/WV0LuqlE4O6iz0v2s6c1vhT+GovHBB9KB63Fd4ubP30fkUGEGxsUknP7C+?= =?us-ascii?Q?WiEBIhD6B+6yY/FOtoKyjTSCSWUOg/F4gH9OB5yu4KMGOngWyXQMGHGdKdBZ?= =?us-ascii?Q?H7Dp/xYf/OYIII+kxttPhZ5SalZFP1TFX6/86c/VlfM/FceuzWiLzl4s13vd?= =?us-ascii?Q?0UYmrd4nbEfmdRLNT0TYNsTPPAS5lqDj41Zk+G2CafzFr3YEo2O4VPx4RP42?= =?us-ascii?Q?15q6eqrSRRmVKdMioY458EaMdRgVBRogAYVTvSxu8Mbn5o9Tcy7eI2U/ot0Y?= =?us-ascii?Q?mdNFj3ZSoTY1yRjId78f+8DSHaG2YYssF/Ri5v5l?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DS0PR11MB6494.namprd11.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 12f18147-7058-4860-44d7-08db7160f7c9 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Jun 2023 07:35:49.0534 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: bZ13RPwX3GLZyNFsk3otgGJNMoGOmoTFWZigxZjzqdntTpWxT6OwBboGh3nEUZNYeYyYo6DBP737UBlUgoNufA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR11MB8334 X-OriginatorOrg: intel.com X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Hi Kumara, Please see replies inline. Thanks, Jiayu > -----Original Message----- > From: Kumara Parameshwaran > Sent: Tuesday, November 1, 2022 3:06 PM > To: Hu, Jiayu > Cc: dev@dpdk.org; Kumara Parameshwaran > ; Kumara Parameshwaran > > Subject: [PATCH v5] gro : fix reordering of packets in GRO library >=20 > From: Kumara Parameshwaran >=20 > When a TCP packet contains flags like PSH it is returned immediately to t= he > application though there might be packets of the same flow in the GRO tab= le. > If PSH flag is set on a segment packets up to the segment should be deliv= ered > immediately. But the current implementation delivers the last arrived pac= ket > with PSH flag set causing re-ordering >=20 > With this patch, if a packet does not contain only ACK flag and if there = are no > previous packets for the flow the packet would be returned immediately, > else will be merged with the previous segment and the flag on the last > segment will be set on the entire segment. > This is the behaviour with linux stack as well. >=20 > Signed-off-by: Kumara Parameshwaran > Co-authored-by: Kumara Parameshwaran > --- > v1: > If the received packet is not a pure ACK packet, we check if > there are any previous packets in the flow, if present we indulge > the received packet also in the coalescing logic and update the flags > of the last recived packet to the entire segment which would avoid > re-ordering. >=20 > Lets say a case where P1(PSH), P2(ACK), P3(ACK) are received in > burst mode, > P1 contains PSH flag and since it does not contain any prior packets in > the flow > we copy it to unprocess_packets and P2(ACK) and P3(ACK) are > merged together. > In the existing case the P2,P3 would be delivered as single segment > first and the > unprocess_packets will be copied later which will cause reordering. > With the patch > copy the unprocess packets first and then the packets from the GRO > table. >=20 > Testing done > The csum test-pmd was modifited to support the following > GET request of 10MB from client to server via test-pmd (static arp > entries added in client > and server). Enable GRO and TSO in test-pmd where the packets > recived from the client mac > would be sent to server mac and vice versa. > In above testing, without the patch the client observerd re-ordering > of 25 packets > and with the patch there were no packet re-ordering observerd. >=20 > v2: > Fix warnings in commit and comment. > Do not consider packet as candidate to merge if it contains SYN/RST > flag. >=20 > v3: > Fix warnings. >=20 > v4: > Rebase with master. >=20 > v5: > Adding co-author email >=20 > lib/gro/gro_tcp4.c | 45 +++++++++++++++++++++++++++++++++++++-------- > lib/gro/rte_gro.c | 18 +++++++++--------- > 2 files changed, 46 insertions(+), 17 deletions(-) >=20 > diff --git a/lib/gro/gro_tcp4.c b/lib/gro/gro_tcp4.c index > 0014096e63..7363c5d540 100644 > --- a/lib/gro/gro_tcp4.c > +++ b/lib/gro/gro_tcp4.c > @@ -188,6 +188,19 @@ update_header(struct gro_tcp4_item *item) > pkt->l2_len); > } >=20 > +static inline void > +update_tcp_hdr_flags(struct rte_tcp_hdr *tcp_hdr, struct rte_mbuf *pkt) > +{ > + struct rte_ether_hdr *eth_hdr; > + struct rte_ipv4_hdr *ipv4_hdr; > + struct rte_tcp_hdr *merged_tcp_hdr; > + > + eth_hdr =3D rte_pktmbuf_mtod(pkt, struct rte_ether_hdr *); > + ipv4_hdr =3D (struct rte_ipv4_hdr *)((char *)eth_hdr + pkt->l2_len); > + merged_tcp_hdr =3D (struct rte_tcp_hdr *)((char *)ipv4_hdr + pkt- > >l3_len); > + merged_tcp_hdr->tcp_flags |=3D tcp_hdr->tcp_flags; } The Linux kernel updates the TCP flag via "tcp_flag_word(th2) |=3D flags & = (TCP_FLAG_FIN | TCP_FLAG_PSH)", which only adds FIN and PSH at most to the merge packet. > + > int32_t > gro_tcp4_reassemble(struct rte_mbuf *pkt, > struct gro_tcp4_tbl *tbl, > @@ -206,6 +219,7 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, > uint32_t i, max_flow_num, remaining_flow_num; > int cmp; > uint8_t find; > + uint32_t start_idx; >=20 > /* > * Don't process the packet whose TCP header length is greater @@ - > 219,13 +233,6 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, > tcp_hdr =3D (struct rte_tcp_hdr *)((char *)ipv4_hdr + pkt->l3_len); > hdr_len =3D pkt->l2_len + pkt->l3_len + pkt->l4_len; >=20 > - /* > - * Don't process the packet which has FIN, SYN, RST, PSH, URG, ECE > - * or CWR set. > - */ > - if (tcp_hdr->tcp_flags !=3D RTE_TCP_ACK_FLAG) > - return -1; > - > /* trim the tail padding bytes */ > ip_tlen =3D rte_be_to_cpu_16(ipv4_hdr->total_length); > if (pkt->pkt_len > (uint32_t)(ip_tlen + pkt->l2_len)) @@ -264,12 > +271,30 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, > if (tbl->flows[i].start_index !=3D INVALID_ARRAY_INDEX) { > if (is_same_tcp4_flow(tbl->flows[i].key, key)) { > find =3D 1; > + start_idx =3D tbl->flows[i].start_index; > break; > } > remaining_flow_num--; > } > } >=20 > + if (tcp_hdr->tcp_flags !=3D RTE_TCP_ACK_FLAG) { > + /* > + * Check and try merging the current TCP segment with the > previous > + * TCP segment if the TCP header does not contain RST and > SYN flag > + * There are cases where the last segment is sent with > FIN|PSH|ACK > + * which should also be considered for merging with previous > segments. > + */ > + if (find && !(tcp_hdr->tcp_flags & > (RTE_TCP_RST_FLAG|RTE_TCP_SYN_FLAG))) > + /* > + * Since PSH flag is set, start time will be set to 0 so it > will be flushed > + * immediately. > + */ > + tbl->items[start_idx].start_time =3D 0; > + else > + return -1; > + } The nested if-else check is not straightforward, and it's hard to read the = condition-action of different combinations of flag bits. In addition, are all flag bits conside= red like Linux kernel? > + > /* > * Fail to find a matched flow. Insert a new flow and store the > * packet into the flow. > @@ -304,8 +329,12 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, > is_atomic); > if (cmp) { > if (merge_two_tcp4_packets(&(tbl->items[cur_idx]), > - pkt, cmp, sent_seq, ip_id, 0)) > + pkt, cmp, sent_seq, ip_id, 0)) > { > + if (tbl->items[cur_idx].start_time =3D=3D 0) > + update_tcp_hdr_flags(tcp_hdr, tbl- > >items[cur_idx].firstseg); > return 1; > + } > + > /* > * Fail to merge the two packets, as the packet > * length is greater than the max value. Store diff --git > a/lib/gro/rte_gro.c b/lib/gro/rte_gro.c index e35399fd42..87c5502dce > 100644 > --- a/lib/gro/rte_gro.c > +++ b/lib/gro/rte_gro.c > @@ -283,10 +283,17 @@ rte_gro_reassemble_burst(struct rte_mbuf **pkts, > if ((nb_after_gro < nb_pkts) > || (unprocess_num < nb_pkts)) { > i =3D 0; > + /* Copy unprocessed packets */ > + if (unprocess_num > 0) { > + memcpy(&pkts[i], unprocess_pkts, > + sizeof(struct rte_mbuf *) * > + unprocess_num); > + i =3D unprocess_num; > + } Why copy unprocess pkts first? This is for avoiding out-of-order? Thanks, Jiayu > /* Flush all packets from the tables */ > if (do_vxlan_tcp_gro) { > - i =3D gro_vxlan_tcp4_tbl_timeout_flush(&vxlan_tcp_tbl, > - 0, pkts, nb_pkts); > + i +=3D > gro_vxlan_tcp4_tbl_timeout_flush(&vxlan_tcp_tbl, > + 0, &pkts[i], nb_pkts - i); > } >=20 > if (do_vxlan_udp_gro) { > @@ -304,13 +311,6 @@ rte_gro_reassemble_burst(struct rte_mbuf **pkts, > i +=3D gro_udp4_tbl_timeout_flush(&udp_tbl, 0, > &pkts[i], nb_pkts - i); > } > - /* Copy unprocessed packets */ > - if (unprocess_num > 0) { > - memcpy(&pkts[i], unprocess_pkts, > - sizeof(struct rte_mbuf *) * > - unprocess_num); > - } > - nb_after_gro =3D i + unprocess_num; > } >=20 > return nb_after_gro; > -- > 2.25.1