From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 28F4B7D06 for ; Thu, 14 Dec 2017 03:45:36 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 13 Dec 2017 18:45:35 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.45,399,1508828400"; d="scan'208";a="2649785" Received: from dpdk15.sh.intel.com ([10.67.111.77]) by orsmga006.jf.intel.com with ESMTP; 13 Dec 2017 18:45:33 -0800 From: Jiayu Hu To: dev@dpdk.org Cc: konstantin.ananyev@intel.com, jianfeng.tan@intel.com, junjie.j.chen@intel.com, stephen@networkplumber.org, john.mcnamara@intel.com, matvejchikov@gmail.com, Jiayu Hu Date: Thu, 14 Dec 2017 10:49:38 +0800 Message-Id: <1513219779-100115-2-git-send-email-jiayu.hu@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1513219779-100115-1-git-send-email-jiayu.hu@intel.com> References: <1511579860-37020-1-git-send-email-jiayu.hu@intel.com> <1513219779-100115-1-git-send-email-jiayu.hu@intel.com> Subject: [dpdk-dev] [PATCH v2 1/2] gro: code cleanup X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Dec 2017 02:45:37 -0000 This patch updates gro codes as follows: - extract common internal functions for supporting tunneled gro - rename internal functions and variants for better understanding - update the programmer guide and function comments - comply RFC 6864 to process IP ID fields Signed-off-by: Jiayu Hu --- .../prog_guide/generic_receive_offload_lib.rst | 246 +++++++++------- doc/guides/prog_guide/img/gro-key-algorithm.png | Bin 0 -> 28231 bytes lib/librte_gro/gro_tcp4.c | 324 +++++++-------------- lib/librte_gro/gro_tcp4.h | 251 +++++++++++----- lib/librte_gro/rte_gro.c | 73 +++-- lib/librte_gro/rte_gro.h | 90 +++--- 6 files changed, 501 insertions(+), 483 deletions(-) create mode 100644 doc/guides/prog_guide/img/gro-key-algorithm.png diff --git a/doc/guides/prog_guide/generic_receive_offload_lib.rst b/doc/guides/prog_guide/generic_receive_offload_lib.rst index 22e50ec..f07d8f6 100644 --- a/doc/guides/prog_guide/generic_receive_offload_lib.rst +++ b/doc/guides/prog_guide/generic_receive_offload_lib.rst @@ -32,128 +32,162 @@ Generic Receive Offload Library =============================== Generic Receive Offload (GRO) is a widely used SW-based offloading -technique to reduce per-packet processing overhead. It gains performance -by reassembling small packets into large ones. To enable more flexibility -to applications, DPDK implements GRO as a standalone library. Applications -explicitly use the GRO library to merge small packets into large ones. - -The GRO library assumes all input packets have correct checksums. In -addition, the GRO library doesn't re-calculate checksums for merged -packets. If input packets are IP fragmented, the GRO library assumes -they are complete packets (i.e. with L4 headers). - -Currently, the GRO library implements TCP/IPv4 packet reassembly. - -Reassembly Modes ----------------- - -The GRO library provides two reassembly modes: lightweight and -heavyweight mode. If applications want to merge packets in a simple way, -they can use the lightweight mode API. If applications want more -fine-grained controls, they can choose the heavyweight mode API. - -Lightweight Mode -~~~~~~~~~~~~~~~~ - -The ``rte_gro_reassemble_burst()`` function is used for reassembly in -lightweight mode. It tries to merge N input packets at a time, where -N should be less than or equal to ``RTE_GRO_MAX_BURST_ITEM_NUM``. - -In each invocation, ``rte_gro_reassemble_burst()`` allocates temporary -reassembly tables for the desired GRO types. Note that the reassembly -table is a table structure used to reassemble packets and different GRO -types (e.g. TCP/IPv4 GRO and TCP/IPv6 GRO) have different reassembly table -structures. The ``rte_gro_reassemble_burst()`` function uses the reassembly -tables to merge the N input packets. - -For applications, performing GRO in lightweight mode is simple. They -just need to invoke ``rte_gro_reassemble_burst()``. Applications can get -GROed packets as soon as ``rte_gro_reassemble_burst()`` returns. - -Heavyweight Mode -~~~~~~~~~~~~~~~~ - -The ``rte_gro_reassemble()`` function is used for reassembly in heavyweight -mode. Compared with the lightweight mode, performing GRO in heavyweight mode -is relatively complicated. - -Before performing GRO, applications need to create a GRO context object -by calling ``rte_gro_ctx_create()``. A GRO context object holds the -reassembly tables of desired GRO types. Note that all update/lookup -operations on the context object are not thread safe. So if different -processes or threads want to access the same context object simultaneously, -some external syncing mechanisms must be used. - -Once the GRO context is created, applications can then use the -``rte_gro_reassemble()`` function to merge packets. In each invocation, -``rte_gro_reassemble()`` tries to merge input packets with the packets -in the reassembly tables. If an input packet is an unsupported GRO type, -or other errors happen (e.g. SYN bit is set), ``rte_gro_reassemble()`` -returns the packet to applications. Otherwise, the input packet is either -merged or inserted into a reassembly table. - -When applications want to get GRO processed packets, they need to use -``rte_gro_timeout_flush()`` to flush them from the tables manually. +technique to reduce per-packet processing overheads. By reassembling +small packets into larger ones, GRO enables applications to process +fewer large packets directly, thus reducing the number of packets to +be processed. To benefit DPDK-based applications, like Open vSwitch, +DPDK also provides own GRO implementation. In DPDK, GRO is implemented +as a standalone library. Applications explicitly use the GRO library to +reassemble packets. + +Overview +-------- + +In the GRO library, there are many GRO types which are defined by packet +types. One GRO type is in charge of process one kind of packets. For +example, TCP/IPv4 GRO processes TCP/IPv4 packets. + +Each GRO type has a reassembly function, which defines own algorithm and +table structure to reassemble packets. We assign input packets to the +corresponding GRO functions by MBUF->packet_type. + +The GRO library doesn't check if input packets have correct checksums and +doesn't re-calculate checksums for merged packets. The GRO library +assumes the packets are complete (i.e., MF==0 && frag_off==0), when IP +fragmentation is possible (i.e., DF==0). Additionally, it complies RFC +6864 to process the IPv4 ID field. -TCP/IPv4 GRO ------------- +Currently, the GRO library provides GRO supports for TCP/IPv4 packets. + +Two Sets of API +--------------- + +For different usage scenarios, the GRO library provides two sets of API. +The one is called the lightweight mode API, which enables applications to +merge a small number of packets rapidly; the other is called the +heavyweight mode API, which provides fine-grained controls to +applications and supports to merge a large number of packets. + +Lightweight Mode API +~~~~~~~~~~~~~~~~~~~~ + +The lightweight mode only has one function ``rte_gro_reassemble_burst()``, +which process N packets at a time. Using the lightweight mode API to +merge packets is very simple. Calling ``rte_gro_reassemble_burst()`` is +enough. The GROed packets are returned to applications as soon as it +finishes. + +In ``rte_gro_reassemble_burst()``, table structures of different GRO +types are allocated in the stack. This design simplifies applications' +operations. However, limited by the stack size, the maximum number of +packets that ``rte_gro_reassemble_burst()`` can process in an invocation +should be less than or equal to ``RTE_GRO_MAX_BURST_ITEM_NUM``. + +Heavyweight Mode API +~~~~~~~~~~~~~~~~~~~~ + +Compared with the lightweight mode, using the heavyweight mode API is +relatively complex. Firstly, applications need to create a GRO context +by ``rte_gro_ctx_create()``. ``rte_gro_ctx_create()`` allocates tables +structures in the heap and stores their pointers in the GRO context. +Secondly, applications use ``rte_gro_reassemble()`` to merge packets. +If input packets have invalid parameters, ``rte_gro_reassemble()`` +returns them to applications. For example, packets of unsupported GRO +types or TCP SYN packets are returned. Otherwise, the input packets are +either merged with the existed packets in the tables or inserted into the +tables. Finally, applications use ``rte_gro_timeout_flush()`` to flush +packets from the tables, when they want to get the GROed packets. + +Note that all update/lookup operations on the GRO context are not thread +safe. So if different processes or threads want to access the same +context object simultaneously, some external syncing mechanisms must be +used. + +Reassembly Algorithm +-------------------- + +The reassembly algorithm is used for reassembling packets. In the GRO +library, different GRO types can use different algorithms. In this +section, we will introduce an algorithm, which is used by TCP/IPv4 GRO. -TCP/IPv4 GRO supports merging small TCP/IPv4 packets into large ones, -using a table structure called the TCP/IPv4 reassembly table. +Challenges +~~~~~~~~~~ -TCP/IPv4 Reassembly Table -~~~~~~~~~~~~~~~~~~~~~~~~~ +The reassembly algorithm determines the efficiency of GRO. There are two +challenges in the algorithm design: -A TCP/IPv4 reassembly table includes a "key" array and an "item" array. -The key array keeps the criteria to merge packets and the item array -keeps the packet information. +- a high cost algorithm/implementation would cause packet dropping in a + high speed network. -Each key in the key array points to an item group, which consists of -packets which have the same criteria values but can't be merged. A key -in the key array includes two parts: +- packet reordering makes it hard to merge packets. For example, Linux + GRO fails to merge packets when encounters packet reordering. -* ``criteria``: the criteria to merge packets. If two packets can be - merged, they must have the same criteria values. +The above two challenges require our algorithm is: -* ``start_index``: the item array index of the first packet in the item - group. +- lightweight enough to scale fast networking speed -Each element in the item array keeps the information of a packet. An item -in the item array mainly includes three parts: +- capable of handling packet reordering -* ``firstseg``: the mbuf address of the first segment of the packet. +In DPDK GRO, we use a key-based algorithm to address the two challenges. -* ``lastseg``: the mbuf address of the last segment of the packet. +Key-based Reassembly Algorithm +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +:numref:`figure_gro-key-algorithm` illustrates the procedure of the +key-based algorithm. Packets are classified into "flows" by some header +fields (we call them as "key"). To process an input packet, the algorithm +searches for a matched "flow" (i.e., the same value of key) for the +packet first, then checks all packets in the "flow" and tries to find a +"neighbor" for it. If find a "neighbor", merge the two packets together. +If can't find a "neighbor", store the packet into its "flow". If can't +find a matched "flow", insert a new "flow" and store the packet into the +"flow". + +.. note:: + Packets in the same "flow" that can't merge are always caused + by packet reordering. + +The key-based algorithm has two characters: + +- classifying packets into "flows" to accelerate packet aggregation is + simple (address challenge 1). + +- storing out-of-order packets makes it possible to merge later (address + challenge 2). + +.. _figure_gro-key-algorithm: + +.. figure:: img/gro-key-algorithm.* + :align: center + + Key-based Reassembly Algorithm + +TCP/IPv4 GRO +------------ -* ``next_pkt_index``: the item array index of the next packet in the same - item group. TCP/IPv4 GRO uses ``next_pkt_index`` to chain the packets - that have the same criteria value but can't be merged together. +The table structure used by TCP/IPv4 GRO contains two arrays: flow array +and item array. The flow array keeps flow information, and the item array +keeps packet information. -Procedure to Reassemble a Packet -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Header fields used to define a TCP/IPv4 flow include: -To reassemble an incoming packet needs three steps: +- source and destination: Ethernet and IP address, TCP port -#. Check if the packet should be processed. Packets with one of the - following properties aren't processed and are returned immediately: +- TCP acknowledge number - * FIN, SYN, RST, URG, PSH, ECE or CWR bit is set. +TCP/IPv4 packets whose FIN, SYN, RST, URG, PSH, ECE or CWR bit is set +won't be processed. - * L4 payload length is 0. +Header fields deciding if two packets are neighbors include: -#. Traverse the key array to find a key which has the same criteria - value with the incoming packet. If found, go to the next step. - Otherwise, insert a new key and a new item for the packet. +- TCP sequence number -#. Locate the first packet in the item group via ``start_index``. Then - traverse all packets in the item group via ``next_pkt_index``. If a - packet is found which can be merged with the incoming one, merge them - together. If one isn't found, insert the packet into this item group. - Note that to merge two packets is to link them together via mbuf's - ``next`` field. +- IP ID. The IP ID fields of the packets, whose DF bit is 0, should be + increased by 1. -When packets are flushed from the reassembly table, TCP/IPv4 GRO updates -packet header fields for the merged packets. Note that before reassembling -the packet, TCP/IPv4 GRO doesn't check if the checksums of packets are -correct. Also, TCP/IPv4 GRO doesn't re-calculate checksums for merged -packets. +.. note:: + We comply RFC 6864 to process the IP ID field. Specifically, + we only check IP ID fields for the packets whose DF bit is 0. + For the packets whose DF bit is 1, we don't check IP ID fields. + Additionally, packets whose values of DF bit are different can't + be merged. diff --git a/doc/guides/prog_guide/img/gro-key-algorithm.png b/doc/guides/prog_guide/img/gro-key-algorithm.png new file mode 100644 index 0000000000000000000000000000000000000000..89cf427b3c7c406d7ddfb485d3cc5122b2206c1f GIT binary patch literal 28231 zcmd?Qc|4n2`~RD^yTjJ%q@_x!c9)i-F;k)~rGufZA&SsSwWd%*43R{42Q_QADn%)3 zN=mjCF-8XxLy0M-ASDP%QA2`6_}yvu^L?J*bAIQ%&iVWN(N`L|?{%;1TG#bi*IMg+ zM_;ot7Z*D$1_FV^FJCgf0RnC2f1WuMwXd~ogQuRf$b{D1htpEu+@bL@2t7{ zjdfZvaRk+t%3i*jqE_sJo^M2ELh2S;R5%O1^+KMv`%ow|9v?ne7lP!nX~;}xzpE_- zVOf3ajVdqdBGtIQnS1w?YikeQf>>l{W=CiV|6WSb(H}X@Z4A>a2z&G=U~V?^V7^>@ zjz@|6ix1Lm{V#}rB+16#f3j=V#@eWSz0RF&XH{G3k*x#0;pqO>y(Kwc<`G-*qV6Aq z>b=i9NwhZ5FZa>KY!c0$Th>8o?oIAixuaZmow!$PzqyGY)3?ZQ=3eeBU)d(BwG*pB zf7w-@wOEL_IilNwxi(x{Td23YdQ%kh&AJD7WV%@6kBNOheGmJr`sVr_8}&OLH7=#s zU1X+fZ>T=<{xZUiiCGxXi}hhYeR;g=WzhI0-l4w z=H7Wzj)+otS*x6BS=Ty^ayw(HY~n7iy+luVGzcwc+(sqIi|;Ux z*daJK@FVQV+fqflax0M1sAgShdd_5q&Bjj`c&L3Ic+>d%FG;BS z{oW%shA<|nDLK1q~F)^@c;Ex58wa4-4LWVJyeZeU6_;r#wlrk`?fY0_1Yhw z+202~^!;6wNC#Y7W$F9410r%z$22f(lG!zLBthCMs&j3qf6`8cGD47*XqcwP0R@kJ zWs6j6>zf^j-_rla2(}s(&*UKHL%0_85K5TVZb(s!J zW1djE&{AGT_p8SshCY!tlzx#c#}Ukfe$95aQhf`6NG!@2lgvYhI99d2Xl=R54$EV*kZW_!yQ4taA~|w3e`ONv&Enww|p0= zmrvzT9*BJ3nwbikxbD#3yVpUWt7BhZY2pJ}?ZxoXEqe_APKYPe@dg!dB&esOgg%$2 zb;e8ei#%MZSec*N!0-pE$qp3KbP$&5GYhrcQ~k<-DxflqkJtbFD|;5 z2h6YeMUP(bl`=!e#5d{|hZ*xc8WsI*JG6coFMm@wz>4tXmMa1O%etqw1$0`hk+iW+ z&vKxAj+nPre+SQHzLh*j`td~FCud4;u(#)`O;~{oX6*MfAXynDp=qx{R8e0{(pla} zNPQr;(R?E;%zq2$K`ZCbDuy+JRP}8sOvr;-)YS{~%MtG|=kcu7Qa*iZ=XPnzq-d_imMArPqcxk}c`H-lj1NbHn}{GcgKSQcB9k}x06 z4ih!-?QK?*V{RIX&`~e^>I8M=f5o_)Zvwq(%UYYH5=e_j{G2FV+vU;sC14?aCHXF0 zka^<{F(IqJce(Fs@q6Au>j#@|wa6|+ztc`D>Jy7OTx6%X%Lw#wwy8h7aeba5&oCXB zh*A0oIwQ4v-n92{V-H7o)Of$$$6|v`AkdfV-=G$>+(D=-1}T9^QOX2u`8ae)05ld; z)F-c6bQ|?YiFn*bg@nbsoPj$LkNEjYQ`6 z8$s)9tH=m$vL6j7Tfat^W;VI8H-QplDFc|G4dJq@3X4s{;X!+_MD~#QgbCr=yxL)IGD<)7L?nfx6Oe{J)r|^qy7^daDCs*6Vae_P zk6X5M#^B7qnoYyAqMx8%KJvN5Cj=i|3 z>8irLIdTFe7yhd9zD{zOs^0;+55X33)^g}2=5dkyIjtpB-QB30WmP?pr#ie;RMh?3 zujLIg-JsBsRb99ji+X;pl zUZ?Z7<;z*gq&u!k`6VSjH}BkR!_!(g4cUJDO8TRszRzhD6htaNekcm}XW+RcSdt57 zs5gUi_nZp(sY87HEw;kvCrGSb`z#J&QCAUG4!NobGacw?9fEcdXQ%GoO6RE%3dwF> zx7ed)rI!<3#`}y{B{&Gj;>55gM%HGYfoHfQa>Uzg-GJk`ZS9-37ll4Ot61%M?Z|xc zkzpgj9^Yh&gvPcO!^@iOQk}y^a`lbXK@051j7hQy5|6>|R|v{cxwMkNpf8a28S?|B zM+(F=6kW7>b}wHP-j{bySoR;l*uu~Brn(Q!RjSIf4BPodsJfPC{sLuN`*Nxp<(5qF zYX*&npunv{zX#JQ@?98lHqChM%MB1QnNiK{bs~DzFT?bTRwZY2Mg2GlJu`3Sk5v$$H2qpseC!KhU{m(rUQd-$~u03 z3k_QeH=Oqgb!tgp}7Qn@@L++f`=N5(W#GWF5w$*-I-` zd!$2Hy70K0JBh}=v5LJaDhIu6v~}!Ua^J&8FbSWomCcnbI(EaW>!f{y+h9wm24FJNePml zAuj8Vo7swI7-%f8)8}j{e;k&Rktry0u{2~{>qnPw&a-~+GZd#(^6V59>G@lgCnsMKXG|)GYTX7E1gF|-2t``NI-=dU~ADwGJ(3(f=5}(b|GZS(G<Vc+h^U_GqammjUn5bqzM!hAI3Co?Lcdg0~y) zu3=xx$4*@_%3p88%m+6nc%U`X7-6pa;J4o%{7_bhm+O?$TVD8F_o-c2DvDW!fJm}szWE=0w8HeAhwDM7zE3Qt`& z;?3)8`3fL|esg|J@r8%=kcOX}{`7?gy~jISepEKvFMh-IWAQ_l4sz8>f4gS+d8KR* zX?&FR>Ft}CH9lM=U%$}@GSDp97d3Hr0Rty&nHuC&?kHT6*!W-&sqj5LW!5PfLIzL4 z->n+;hJM|Y_@%_)MbYwEF3c5q>UHl~ZkkNym$ZAyod#O8Dmh~KItLvcyTyPt#Phio zG~y9%5sPVK#kK@sX%f_Xjc4Lx#Uqe%RmB&OD+X_Z8_=-QdpmRxUa>7veKOor`Qbxj z>zGQ6Ni(x>9ToTV8Ra~s#Z!NqzV*LZQ#dc@b0TXD*ktoNR`(gT^#bQX0otJ zDPU)x%v>eRtPd7y7t%0e^PoxU1XyIh!I%z}DEm$Wv-sYd6z9K--ygi@ICI3?aPk3P zDVUaRQP;CPql?YLrt{85XiJ*Rhh;`b^8P4n=T^eXCQIUD7RWQ1i(xfmuPPcVCEKp` zg%Qv?cUFgxPEO$Nvi1puQ1Cg#l*XOTm1t$_okaC986$3M>Cm`$=GZN(M0g$p*?nMf z%0b^qE;i$(;Dp1^HGzzl#>F%-q`9NBj~9c|A7tU4aJV>E{`SxnM>p zUbW9|OM#Wjo7q_F80F#|YFTA%=;lKR#uJB^TSj%HOeUzzURt}@>u*)E#5`@u9NgKl zV7=6PFDiDL2u=(|R<=>9@H3xR9pN3?W+fq6@D6htQO!{({V&rMEbHnET=4U>Ek41j|Tng1lZ&Co{^EhOs|Nn6vwunIj}1H!v|`TIWc=~lBB|0x~(Y{?-=Xx z4Ywp2W;KpQIa?a)=ojM?JNsD%OI24E51G4FXIVs*q<;WMA)16=XO?&+rLj{hWVDTJ zmdPk@E?3A~kNi2+eU!-ap19>?sE;p;fpm{8DCg$S)a-2yIQp)|FMDnE?X1_mCyz`| zQxMZ)g?7VasYmwJ*Z68anh?3zN=t#a6O{3Iv7GPDhR{9u4eTe2avHhBMqQdq)sryzJg{(7bh%ukZgZilGCWGAg{F(ZJBA_@~#` zhuR?v8HMBqYn8|RnHj8Cm1_0P=!qJFl}#MC&6X^t(?aPA{1h54msO%K z9WOXQVO^^f3OwrT>J@ZzyuQia-`lU)zf_8NR=y9h^-2o#wW(Dro%kuSE^?b!)-CW+ zvBZgv8jtPKp}Mmg)q^)^s=$=H<76F25fKOGte`3K92JZbbRFz&l&X`{Tn29Mj*eUAy^#Y4gEgtHdQ4z zeK+r&(m9(^XEawMUW$GWO=fzlEF1htU(q$pz38!h9&#Hm|awxJmP3AWkTMI#~cM=)#)lUGu;pNXXvz~zCY!hSz)Av_m!!TWRQ*p$_c zBztm0SeY0O?IOKj)rNV{COgSg>2Yh<km&@OkzKT~GkuO!pP1=e|olUKE<7uq8#>hNpV!j;85uOgA&Cj)y2r#mJS&Yu*wKA-%KiF6?fOze(YPhd*t|}I!D49~ z^SGNMsaKgme-~B=nee{gZ6&w3A}h8Gm%q&pIj4kE^vfI)i!{jA9>58ewK|bqrjPP2 z%eP*~433Ds@HK~m_8BM~co6^_bVU}a(y-DKb^>(3Bo8Ntn5*%&UpI{8zq3oVcanVp zM6GiIw(qiB>s*d<;5%?@CBxr|;jh!qg4~UadpK3`0ye45Wu(J3k z2R$>x)7$dekINj%JF6%d(%`?Fxze|Kpe>P?5*=dHlGJvc7AclojK;8hc5(}7RzNJ3 z3l1;OeVNg;lUuf%T29Rmm4x4vZ=uKJ4rVS+8qUI1gx4B^C*aZ)^M*16MTy$o1Wu%8 z-*@Hh%a=oyUZ!p;c>;~SvkM;t>fcQn_@_Jlzlje6QRPus^VLShiryd6i86qeXS=!D z5U>-pA8~ORe2t*BV}dkPwhq})^@VxZ;`%;DLsZK*a`IeKZDzGqX?&3sJMcU7N#gAh z-N1Luz>`BMz@qQzdA$zaR!(T?yx4xw_ou)r{in8S`#g;FrRurr^u(0MP+W>%D9puD zYan4hC;;X2QSfLOnF>FS`YfXieX^eM4*UWw&UTC-rzjxL}9P z9?<>qxP9Lf&!e_dFr5;S#%o`Uhu+p+Z?}ox3i>GPKSVeGw}pQdrM!w5i&oNc^74mm zdXNVed29p%Sqq zvs|Bjgzr4Txd8vUSK`;tgSw(Zg;e5_rmqw`d`-R9c-4eazrGz-rE0UPUdu0ldl9hY znOTY&Tuc~>PC53U17>o0&98Kf8o^<@lN7!-_O!-%*ZYh_2$PKG6_Zqg;0HLGI4PSw z^l$U9pHm75G}&~>U(ROFiB!IBx5nI{{i_85cI~qaJC)8pLCL&gP%lMNRrAAE(fLoUmCNGnNgOujICtT2l(_$pX!uYHZT~ zuV!~AJbcJ%_H0k~c}vuOGi$zEgS0aFO-jAxSpGzX#l*uCi`nUQhQWZc_{+pSWi|hk zfB~|F&!*=F)hQWOqgZ>-gk+oO=YgYTlwmFvGi#2P6HS1>J$t@@mzVR)rVhFkFO1U^ zq)XPs;vtkipRp$=`VKBj7>q8h^iQMSy*pUo@YM2ttV3qUSq@718NC>_G=7vtXqxF- z_kwaMMSSPv&d7Jvo-Dw>P~iF{9^=kC-+dBQ|+4_ep1E8AZ^mQQ-^G)6;nKO z2QJLSBfd^Oyg73(0Hu8wzK_#~?em?APV4u+3!sYUu ziPCws_RSL?5g&HYp@~ zpT_ayx{my9u{)!Bg|ei!(B|lJtZW1R%w?}pS7GM9=@9R5*DU4gl0BtazIgcCK^#fns)_*H2{8*>=)RSlD|FI@ zn%s9a9EnMe!i{wB169VWn?bMtO*{CmN%0+o-bh<%)PcZ%?F zOE5}DfrfsVM4`Zc1;pXb7r$d>HJ<77%mQSuuf><XGI1C_0l4ITe$TJ3WyI zrYA|w)XjLL(W<-Mj_Q zBwFd19GMw{M6vQ0y&tKml743#5&OjUm&Sz#GC^=$>5Zy^ej+*O^w=SC?29Jmp+|P4 zX|Hlk8G)YSe9oKI(V10;@g=>Nt7E$ho+kdv(o%7Q5cG(5CPk({=~x!$LTL%oCFE7w zA@c?8oSpIK zSW%`qbuua3X{_<)rXISJPzu}>ZO@<_OSrw;Vz-0hwst*5*d|r^SUp9t%-!!kW8&cl z=bs#(?Ka4~zp#>e`r@n~uzAp#5v>2bJGD=rm+JHO7;OIdVWn&aa-D8iG^bOonD)19 zL|Hv=eqE7#Y~p&xjbW=khe^Y#IEBy+Wz`BhWO+-7PpiKX`E$vnNg#nPki#}Zi{~F40U%nnh)kNLUuV!IKo{DR&??vZ` zKJELyh4pSgA(8mvPzN!a^IOw-QY(nXhdb&_KWCe^ufiRNNaDM-s^Th*m`88(9}bfm zz+WjRi0&!tQ6;sz(=OCgWX^6CWXQ!oeTY82(@IJ{<(JBzG>~_;daGh4_ZK1_16}K1 z_H;=wR#u;rVLtJoDSVKU+F2bJxO2`aud$+H@TkwrKv`hr&v`w6;WF|F=W}4O2KyjH zZJ?u~0@`00Yp6zl(fyfOLIUUo-|i0oYoy+CGB>Mt6?OPOyN}#Fel|RJ#%|p?!Ybs= zyQcF-uFgLV0N*E8+#72=9|Su}KALbFqE@9^vt%ALF{3$ODt;w#rdVy^K5C`%U&FKd zE#O%3p|Jf^8ThgL?vHE6%$<+Sv8_2G6YPX=W7XNu0(&4H`ObMVtXBUHW`A|ByQ~tv zOfu8k#Imf-4-!0>zq*fz93>Ua`PWpGe_gm{n3=ZSWLI5(u&mY`j<>UqpA&-}%cklovljf8~g{2d5n>^^iny4r@G-6-sMPTO41CvLe;;4|E?1 z#g(CLIy927KF-fu3peA*CuRDT8L2?<{CR5o*F!xE^IqlMi4K<9W>u&6`JuH4)m@`{ zKT2=@EK@4Zvl7J1+Bdu|^0lrRwvZvjVijzIP}$@_(Zb}4O=Bur{LmF&&fZ$T?MN(dyaY!=?G3n(0UPM0*NW7ua=dH)k+h;?NZmGbk9Pfs>I_RZ8mlCqfK&NfLogH?jZlg_?Ry4P5P6T z7G1ENvf1}-cDs;*PZ=DLv8mGaHLax(kcOp-^c|9ss2 zg>v_ou@m15QThI)LF5^;Z`1$Mt;7Kxcn))4 zB21CdFT$n`mYUlT(3&`9;d% z70MD=8P_8kMYE2+f&amihV802s7Js#suIciu!ObT_N~krvHkwx`39p$eppy&RzGNv zc_fwxjU2Hhq?x|2YGZvQMjjLvlyD)KwWaj;$=>gvpD0v*m0&D~4CIqWyj$Gn4${lCn$-{`b!&l?$ z*=>lH%8_1wq;NH~UHaF2@>@qV;lYtdwH?tkv9|+gXuj*6*_@Ape*`9Q zrF*+VtKLQNvw01-PV`77Ki_ie$zXo$tA{Z>nl+|3c>C|UR6A+-KFXQx`9m*z`-`Yo zV&P*D@wy*{yXNTZ4~Y{MQkr_w=(S&CcIr9YHC9&M6uP!o15+BOcEH$eP4@6k_gdMj zPdn~;I017>8v@aIiHU~6^GJmmp+iwe~j^4xB6RAXTAd6eZ%JW{Rq6-^x5xWyL?a0|~%^8NrA^+NZc+G?`XQnG^9Na+Zy^?JB6d{}!{?&^$_>Teg}j2t_{^x4e7 z2KWb{6a|)u?T0>LucXnD7q?x+Cv{G(Oy6~%YTRq1URK{SFzJ~op5*!JLaxD`z3}wM zY}NmM}HLFEGO zZ_bo5S|-UKTFnY{Z#Pn3%;>y_m`m3zspo{YizjwM1_jFglV4vUkCS-g$K8(u*nu&y z0JixH*rwe2u@v0kyk!YkI5L2%W{0Yf$JYgZ5jj4hGyZlc0aBSzs>?txDze*MNtJfY zP4|6nv2X8^UFLO;F_ECfQ&d0PGtUJi3jk_ zHhTmM3n}t2szPZZPlqeNvBF^_^9hiZ<(A0J0HA?pX9=4O$Rhypm0ox@VtShyX1$w& zEtUP}W{qBnfjUd!2P&k-x2dL{Ipc03;wMNfBV7XgyM(o2X>|3Xl=Xw#SAhPxl9pZx zdILQeBKCpb=Ms)Uc3bN1d$0*)Mkb6*?fjjz&L#Xe*>E)x!spkVV}#7Y+=0!6TjA^L ztIH*Dj)>Zcu4vu+KcvI~8^Xaq>bAe#23u{|-nwlI=;M7z#Iy+~74=12_nixdOS3G4 zMgF8OZhuvJpPP!B2FMdvUAC&9e_yOGGg#~en3f(YmiHn3WXz6lh3F46^TTe<|77xXJut63Xv+V-GIlLt^I zmUTf?qvdxpg$KhogMr>0_*Qp;;DGD9_VV=qpedb}b>j!a=hx>*?n2TvngW0o3143^ zet6I^+qF8Ih&Ea+30YH(GXi~mXkzr-D5@x>?DLp;!C2iKmc>qzCspzvwHgyq>mkPF z>vK2l04QfN&Mj-bGm4br;0mAgGun&#t)?ers-{6iy6k+4J{Eokpbv#_M34+7d*R|v z5NITd#61`BM|qiF)qt6(`=h88Su3Nt$+$p8@u07MJrh1*vAs}~w zZk;;GJGD(&QdyB59-Xq3n8*8xtcYa*@gM;fQOF&KL_mzi^Q6#Yu_EqMOJC%~gye42 zukQe(D1wf28>eS$=3F)!^+hOQhciGW#wN)pidf1WZ5Q=q%JhEGh+4X%J{@Dfn8l8v z#~;qeO8nw+10~wGsapLa727QTKi^=^^2x;T21NXEwGNDb3RWHDDO?3$3&#k^TU+Wx zYl?1`l|7p!f{a(o97Rm>KL~U>YQ9mbk522A@vHT(OB_U>HnXE#LqsD&1XkWYxjhMR zX_cQKCO9al-OTKV2)f%)TU1sBHecSfHaS5FF0kQ<4Pm-9Q?ntIb-x_+KMj8(x>_qTp?#pq zN3|2+c^@aUQZ4J=>IkA^^Qgc4l0A>qYA5GwNpzojNl9!tE4F1sjT}9DKlI?m4+0SAap*rCw1}+&90*mm*L}URn)&+Mz1?^@0{xj z&ZG1_%CW`z)Btp=h_=phNpY1nK!nNu?EYjGCMURvZvSUm!I-+m(x+t#R+c@dgL`GdlcJyJGoyuF+scXeqk4Q z6f@C$&d+8<=dc$W(=9 z(z=4lAn}Gz(%FW`drncw5tqW~9#KUEd#Xy&9(Jr9_bT&b_w0JZ!O+WsxM7OP6-?AW zO&~>0NH?d>CWjL)5#Q3dq=3BaiOVJ}bbAwvy0}V+AUXY|PfwkJ-xm4hZg#D`Fwf_k zli5aU9*NQ+HbFW#fnXHPsZb-sewoMMzD6}ORdNYX|1pN*C5_cVYvd%R;3!u$;o#3F^pqd6 z@q09^QKj)o<8M1*GQXu;tZLU>$3jIXFmtZb{T%UJIf_>v3`l~-b?ltWG<X!4P!3bH!&8Ct(|Fj(HF0h3UxRNd3N3Y5Uid}t@DueU8>9GZ z`d1`omtTxm{Cqi~?P9D5d47I9xzkQAo9a^RJJa`<%k@(K;4qCLomTowT~m50>N4Uw zmS~Nz%c>z$$#ss6DP=M1l zP|Yc+>CI}KAu>oKy?=H6B-U2_YUjx`;WDEhwZ6*I)hC@|MHi|8mWUu)KxHj-xG8Ed z$6*ww7>|b%d~euu816kxovV|>o+8W_MUPb{g%;O($=i#FV?B`wXTNGzblx^RX~-_6 z$T<~UoZfbhjyo83Inu>_(hmEp?^ZoMDpSTw9VQ$?=Y+{}5cr(SP0r9Q;Fe91Kb%kWVo-5+TgJcaV|C_+Y{uhCZ zakpV;3=TwjPq<{^?SvV~Pd3J~TpTE7l*_6Q(4#wD`%OEC@7OSI z=!{`ji(5WI=|GsguHAi?T|V~qocf4hHl&2KGG(l;?h69l4?A+aer~9yq|vetSBbCC z=Drj4WeK_0W@BglkDZ*O|F1yYnX=-dtS@J|B0d}io^C%o=Ug|3nRBXde;d3Bl%d5> zzb>399D6?Y*I4&uk06`5gM4CE7>9i|tBO$Tu?ci8-FUSv%N4^qJ5rI5w+FKFO|g1+ z9{|cN5E*6moH}YLMiB3^*b_2`7YBhTEvt(xXrl4@iekVKGl2Nr3e|STTO0xoZ-$?j zF2@Kfq#go*u~yz((T3l~OntHXUpS}Q#Tf>PUnF0L;hs%gTVh+w3mR!xH@J*srWokH z$AO3?Vs4B%->GYE@O0OX#^Qo3EJ~u8O!EL}z5Z{;F*FrjJ0+m9(?cCJ2irv$3fOpp zZw{h97*K?`@U=Z<8ms(0A5^=shx1(`E}@5!eZ?Fa!v`~)vcU*g1A!8@3Wr~eC6B!v z>jO?1sw_=qtaD$XFo?V@r{_Ahfa1Og>IE;6KA4r!r@EcN0}JJ5qN;XBpA0BJs3QB|*GjP^eQp9465%N|(fnt3T|BLDDl(1W6E zsQ-b!_Sa{LsG0IUolf77csI7X3tIRvwLMW6DRXWR8I26DsZ1f*HSlT3wpu!j58IctKn0tF99#e@SSWMb%h^r7=Wl= zhTg<-*^k^GiRyR}aJ=a>e|ZD`sY_n<0Vi!%ZN?2h8tXL9EvyC4%CO zuMY7$K_7cM8E=)_hy5u>(wm6sqUD*yNB>|?6hkmVoL$a>@M0$_K_H9Q8>?Az4wNd) zhOZcKBzI<$?7gS#Mk!ZkASPfOjIT)NM1>mg5<@|6mWYyVM(P#$mJ)ocr}L5Nu6)8q ziL4ah(nS%3PpQ(p*Y0zS;VVLfEkKlkY|%Rz>|Q57n5fL_OU)@_@qzQnTl7P&zuWA& z|9EJbDb>sBNnHELxGAKi$f+%UeBZD0jR6#Kxv6Z}6W|c@PrV3Oi*1{@>N0oRa>0~9 zDXf5jT@j~15x50JYsuo|vqglf8-0w27{pa(>$|o+g@48z#+6!dOGKUDbr0yGXF$G- zVt2k~?u>>h4I4OiO;IJ~?A-xMMyG|te=xQvJ;N<~YAV`V%B z#T&Rsn27hN>0avYx>twG?F%fcCP0JBEayf!A1a^#{7h!9jfv{l`my6c`Ksb;8*2gE zd2BYRV-3IXb*t#Orz+$tYgkr)#y>#-FjS>c6U%txKwT~@)?!Qq zDoA=Bf> z$p&E9tgQ9RIKSO+*CS^V8v7I50QX;8M4a|{x4^3^o(x@k3ZSj(Wsa_wCxyV|M@OTE zzYDg#`O6n{=X>(K_I#JVS867HwU_osPX{IvXYVCXriS}~-DiIu&mmY46|OL?tSC`s z!Qdlme87E5`_G7lr=$(>KOMIDHmRJg{Ej^X5w)ncWQ*7-Sp`phM1FbTdz(U~x zX?iZHdV7L)d#smDnhp7-hZ$THozlWnsjhmb|I1R8Ol1`GlH-yz$_DnPW|zei>%V*+ zTWIapP_)aG)G;i6uX5+Hv#Y!s(V)}K=|glExD%q2)Y(s|P#zYDgD)uexqBs-;kqZ- z_jKIhqr5!|X*an3Q)|s+NPZ*M@9e#?)CPE`d4?|iyHc4Sgk(bf1nh;IH@v5ubDvzz zrIuI+o`z_5uPr9Upug{aLf5bzh`Xx#Z*&{S4|y(t#W#EAtbVJAR#-ebNwxKd&YR9_ zjdZX>5Usi&6*CEGUs_mIvQ+O}{kpcKF$FyxXyH%>6>Me9lmM4Q!y?N*!4JYqczynz ztBy08za_?E{o<1chgc@Zs7wJ+$xCv}Q7ub{Dkhgo;4`%c?VOZx{+*=3 z$-t9|1S}3wJg+BdC@Js&gw9swUU74Z<+I&oO0N|OEipA-;tN`H5~P|x)E?9O{`y-R{Vc(77SryUyqh$!BuUFaPk2f;6+uPtN#TJdw=^=1a@}}wEt=UqSn_dtCagV_)PP|Ej=dfEHQlQ| zbiUKNZ7x-gb#0`6&M<}*FjPXqs}a2|Y_XZfnGuKY5p{EQPSstJkxBe=%`P9c%f>h; zkA#bms$$SmGZfQeebYNLrqi zRYpS)hfMC<^%_pA3~k4cl+4@JIF+s)x_@@L6PT$w{%n@O8)WAUbj*R<3Qx1HzZ#is zK5iR4MDZ9qoA{$)>5o==yj5H(sm-+2optWD-<^PARcBv_!Z1bQe5ZTpJ7JYJRuyeA zw)M@EEj`a_Yu>z%e~7Z^M#oP3$BnJ|Y;Dqu7+YJ?8D+9cwdS{dE;K3~(NhfD$X-;O zgG&Ftk6Ny0D*bLiD{uOQdkktI?{tBSQF(V^NekBk=&Oa!E1gm6n5om zXn~6!&s2+o`I6oocHBR6V(Xu33t<`5zDse^H0`pTlt+m9b-OeQmGzvgR&gD8OJ}hG z1DET|c92-^^S|cfZyXY>kg&z~DTHMZWJDZ=lI#ud8`I>lIzx!55Q{nI@GLV5=06x2ey&v*)G4mybU?&y#mhWMPsyX+NcM;LFc?e zUkH1Mc%+0H>P;IzHn~X3yrVcL5pSj9P~PNTW0mVaeZ_AyU@>~KVz;)M!M4L8!dVR` zv%jBMy6m<;IO_+baQ1u4gO6n|xuI&W4{b}4Ca$jNpNuZ~c;Hc(tc3i|B-Tamk#XS* z_1Sv?5Whs$w+hzv@|nGbYlac%c@HjhB~|u4f(*FqsRq}CBs5#K^DX%mXR=%7(VA(C zu$|gqzl2krdcVKsfOoT((ZjJUJIJ2lm$LP6>6YB!hN?`?^~~(P2T{v}E;kEr*mk&c zi>Fs2^D5NmI%GMIQ)#QWdj~89XLtb_sy3DOsvBczbGF(ohCj+TwOTH}D4dNw+k#(B zq9{^SCpa<`FlCU-sxtmfKCx1@J&a4#CP5%d`v@VQ09H0hhpV`qrgwdem%Z`;N>|3A z{r`%twcd4rZuQ+6c%6r|B$kp+w|Q*}YgjAR5L{bF43^a5h*#)R$p{?QmeM60kpx^6 zw#!g42Tv$QS6R~f?We=)YZE;uV(B;o%Vfe}@D5w7oQ-^6rRs+n!{O~f2D)pi#ouP1 zh_J4!fHG|S7O*ubhLw|2S1+AaI3*0p*DJ3yxiG}RJZ%sxsJJRZ_GSxKuL`R^I z2uGooi$lvy+4T~OAhPcE;`FDI0B*{TVkRfniTit7nGUHx?>3Z#=^nx~RBAVL#Qx)e z$I5CHeMVaA=hfh%kUGMHFDX>k$+0@nC($oPC;Ug5>DfFgJ_qafm$kAhpKY5~ZxCG<{+l$LP62 zq#6^0*UB=+SO++!G*n6s06n|MfDtA^s2|>1A#aDPaLD%FTQfv@M+@=~+ z;WK>ZA@RJJRyo(OM1U1Q*Yt$%FGRS>TekOb07zycdg4(seSh0x#deKZdfI9IXZRz7 zBVN|!t|~u>KoYfIaz#C9ZDZPQEs|9G@(B-k!L z4P;Y3hSpaSC|$O?h8kOc$qvra4mCnRZc1QXt)2QFAh)_r!xukzRuXWFyeLYTRUu6k z(urTX#zix^&s0VnE30tqhD9Ryc$PP85tkftNg8|?VM0rRhz^w(3u5rkd&4H*6Y~xy zNt%qEYhbz_=_6sya42GxTJY%nfiwap(nW8h!WJ-`D80ufK9)lCa}L(utrR_|$m@J- zfzjITf6Qy*+TBl{o!7=`yN-OV>?(Zp?(7xmdq#smRZo(_zeIy z#El{{)Dw~cVxSA^dgqdCROlL>nMP+^v?H_*Oz4>!W|CcSv8fRcU;MG-B$c!Ewy=L7+4O zkM*|(HTrjLy((C_APEhqY59KKMHab>7MUt(`9ZHI`Y@-EzmIjvqHjJ%u{JgoebFM8 z;y0y#Ybo*)^2qjyC9uDd-C)NeWpyERm%jOOBXV5kgCvAkS`1OFPH3!! ztqi6Rz3C*%S7uQ~btVLEJn2A8VC}^PhoS{7fso|ud09GM>3}F|aC4|TD0YzCbNUVa z&?H9zYQNoh^7ZXX)^bU$+Bi)D`_2nF`peef*Uv=)NjCsTmTA|gbwcmTclJX=eU=2- zRy_``v_w7Qfa<|4)+IB{kF5g@OX#&BhFJ1Jzd!!NXa{V*#!^1D->)IKlY$wN8Val;GT zvUXN>Wr8wdYLy8bi(8GY1Cn^40ojX?C>8N)5hTw;R#4i z=KzdQbdR_7*lp;NIkz&MdyD6IAc>|6JCL&u69K&cTBz* z=sy?o+{l_K&R2EJ7l3s>!tB-m?3=EzBV&&y8YXXl8m-B$$y^C}_I}coBz0M5{#0IM z(s*3xdV#o7iK;q^8#OQzT9kHZM)+33jgbXhdG=(|;yA%lOO@p1j^wHshMj(}f6P

VD+uaKC30<$_JhmKZb=uBVIL9V zS*YKjp8GRj0_JrnbgVf%^~G*-Ow#n$BKA!G4b>oM?{Z$-m1b?nlYWELmgfFyH`nhu zVYA1mXAiNEd4()v?}}~>q^hm*AI4V5CldT`+F!-zB_%He3NwCXiDHXPM3Izh0=Ph4 zu?gXu=r$XB)gANI-2NtB!jZlVbFS-RMq8`;F-kSAgKpG9WW2iqmOJ(vW7c2eWFH0@ zykB!S>*-esV95gvu(Pzr3z6;1)h;-d!_9+eOm(;}byeZSyOcGm1Y%>V$HLm@k3+$14wx7Nh@I+j%RLV3;<V#55dHqwd(&Yc2~(o8}#Je(4aG=OH&~}wCl~g^6w=riY-C@)5w`rux;D@ zHoLL+-9;tCcnCl@xvli`I$>($fus{e(6g6%%>63Mo7vmV1Wa@?4xyStAr#{mp<`bp z<6Id-S&NgMc%WnvW7?WNj;5r+#6+NCi9wq@{F$Fxc6}%3crZ-c2DOL)CeXWA_~7wh z1m7(m-*0kX8rBxtaAZj=dVkjX(kp6oZBi0;Z48%${(6iExeaHi9-Qj2L@itQ&P-wp zQ0BN<0)Glga?m&+98kiYWP)eH+P9Nyp|=D}gO%LI(tN5MFQso&UaKX1lk5qPP?dqf z0viA0p;+LiKZ)p;lR$rKR4-{oIMm!4H^r?ThN7)Hi|uFcg-++Cmd;CVjA85hBm;ny z0zPa1SHBkj@c+0~c4{eh_GUawl#%5&g%e*4=Hd%VZdYtJ4z z>3s6m_ZMy7)=A!6*?)mKD{Ni&vf@wq8-clnR4&8r0JgJ^UPGS<>0OMN6>bm_XrsFO z;1^a#lxs~erJWuTr}eD!oiKkE_I2D|EF5OAfb2`^@E?dQR9Kj{YtwUE_UExvksH*= z<$+*f0Om}T#Os1b-ub^C=_(3h6{8vE*4P)a1DEJ$H zYQi5qz~Fi{e)zlBjX*1lq5E%_=C!xJ#a~>%RlJ=)o2x?ph%zLoDa7wWKCq(-H#?L4 znw^tJRFFn4>nO6%NAe}wFVrLe);WWLnJxJI>p7CmZy~oYmkB5r?@d$^;@G=Mn^_6} zTCP-3WX8;0D!pAN=Nr;;N5MDleCf=@ae)M4OEc|_f2aB`0>1kj0=28HanX5mnYi=# z+TMJ+xyfRMzJD!_sYxf-lMEnnz=|EsXe_SQ6fRO?6e?Wh;GbV35kR)R;4rt*hI#>@4RJ(|XH z?UL@5`6tQE8>;Uxj$`wS%cUu1@?ClittAT!@%ASF7_C? zNPULZp-k$X&TPD?lG+5QKt091H*C5@5B7?gi0eDhE59BsSM*JNf4;7$N)G!n*}Sor zW>9~zi-A^|H;$7*s$U@Ny(01NW{Dk1!WhJpIkQ(@X}l5=LK^g1wPw%T$4RzIj%Bsl zniMbWYxdV9`vS5qkIXF95?Blyu13BT;bY^xwKEW0+v&fke||5z9)&;U-)+jCP}(mt zti9vr>(IolcX0H@nEmr_nDeF{@NxhBi^e?9o%^dUfwU0jUrc&Po>LRz=BU9Xx2jC% zZw&rjom)oNFMH$2B-kWV+f8E>ZP%g@6{oHP8L zlei*I#Ggi1|L*cDSD`LH)af$?q|@GUgi$^-Hl{3`_q`*UF1pQjiVsifh>qxucq7}i zt4gtSuu=7n;^eJ={)QVSz&S(~XNqvJWPT|PK|Q5mhBg5S5fb5=_X}4li`pN?gH#`C zQFCcEzstiy!8!c5X7W#0S6#{u+q?}rqwatFc;$!kyK1NT{@r-9Xi_K>^|HUu?&1pW zsEx1!Ty!vA(7?=#{h3NaW6n_6o%s&5Ak0J01LKQqX~n_N;vzyX`!`P~n3uRfUYsm{ zqJX&h);N?h!R+ynJ%*P#>4zq3&uJa;O{MMcsn?;F98EV3i?jVLiFa^5y7?!XNBk(M z%t8Ca=DtQiLs(4Ce#a+rL9)Wy%QwpJ6AMd3@BYrt{GR=-0`K*vpcEU+9mo%EVIF4O z<4X8l*S570UT;xB#>tMA&J->Litp3V4Q&5=aNaJsYu6%p|3-T*!7uwoPyzXYZ>o?N zqj^7S_^^riP18LS(uo(T#`{1Y#B42;W;@8OwhoI*t%TFL+M0bFka)-*;*)e#3O*gI z#HOn)<-e2BK~4sWcLm8Lc{ zY&c!0m!J|Y8uk1&8G2=Lg(2M0N(DO{Ba(7b_d}4A;;==H@!bcsGU-uY!Ppkh~Qm}3zmP9^_Uwbz{R<+AW0eVHodM%0_Q!X!hY9Y%dFZXsw0=0O?4k4(8I zlkwx#^^b)_g(StbprHn0mjBeBR*|J0dGw^+YZS7cbi}b^QVSkLPNc$G9Gctg!Q=%|Ut)M`5{DR{$RP ziK7%9s>@<(+a4(fQb6E#(}n}J*!WIHZtyX&?8XMfR%xq^StAg+CUNIzAi(@yu-3}~ zWFFo{4e0_AFgT=H7hTWQqC0Jsj-UrDo@>#aZxR4;Zn4!12q<>!aI=_Zl$-W9Pg)}S zP7Var$L4#JlK?0B^sKur5~#PSTZCxVjzo+0@vzek>vgx#Fj)RaY9{)tUB)oDJbb#_ z2)Pj)&C*&aeGLW5b+8+25mW;E&Q6JbeVW^JCC%4X350A}S2qBL7{a{nqwT9;_gHH4M z$-0#>rER@XxV|R8zC=t83WEb7xpm2CcJo;-Kt~jZAfvp+vpEyt4}(SLB7Ey1UR6>0 zO6{5xPf(IKIiB_t%;Ydu_%HdY!PhaNbu&i+M-ofy!d|e-#eIaq-pSh{iaH|#VQP{{IH zyI^PB4{=H1>_#>=mI6;%Sn1;9SEzy8#{XP)F57^)I?iGO#Z;MHE1^69Qlh=r-5%+=cqgOgRz+ClI8NkCEF!u-D>vW z2BEi-GdQVRCDCveyGT2;3_Ekxb?PwrP^T_$+T2fiaQ&Rprtl?GPN$ayH&!vbO7 z6G0O|ix}VsL!%;x@n!9svX{@EEUE7V#EYN-PcVRVcqc4TyyPP{$%X8cUA3u^DRY8& zuL!QNxZM)j^bOCdLGkQ9ZmgN_D~5RS@u%8F=$0OTyhlo9-!xo_!RytU zt48@OuzfDf-v!i;<;nGvdrwK6QebkkTps+2xOe1KRww|^06e(Abcej<58n1-wMhBz zKLFhmBVJw70D^#%#TxA_jZWkATA#y#A+LP;l7t7rJiTdKhZoZ80i758!cf zT2CnsnN9+q+>@2c>(f}PlUYa>t3mEEw|6*YDNqj^@dE98ujMyxfw_fa3x`3$&F~xy z#Zv2$H}Icqu*e9M@PMM^w&ZShcyi0}zGK#gLB+`v>viS>8!yL$r9qeP>NtPT(NrbQ zLf%TT=ak+_J+J8y;fCst*QczX@#EuPw)O-Q+mMuDOGFruAk%9NXrM?YyX!Gjgsqa3 zkLMPx8OP;Q)X#~eR}V2`U(a`p74Xmj*S7iA5L@vWRbK+p*XqekpKfUzx2e`e^t4Y8 zN%ma}lIs;=le*w_+I}q9(o$_8;qjFVIgK}fao{~M~X%5kyOf3a=~qkf_!gi+;*pP3VlBhU+zg0fP2_il98o z?r`pB3Vij%hHDT|Fl9H4H_QRbnAa!`duw@l)@qOSX={^uNEjE={LsGy)h$ANuy=;& z#yjs4bPWj9`rYC(s}`#yK-+>pHJb1?)`vGDf4N?1&>_q~c4mqUKm zw*i$du+L2o7v`k<)_P(rIGHPh^_;fhI%}u5FxZ!l`fo@HJKo!sHD=Hroq&mU&I|%{ zShjA07lYlJfSB5tTgh}Ju8kYD#T|jc$}MK=xsqsB#o-92W-!ZpUjfr>IGg&bPP7Do zDi7%qT{^LF@B!+5)onr~5zY%r67@Re)L}4gF2a8;k@whKwxKZHPZMFn6vg$i3SVJt-_;DV^!in z!}6dKR>cY$oYHYd&Mz=n-Qrov?~Ot6n5Awbbqhf-e8O`LmDn8h$=WYdSq0XDAj_=( zxgutQ1k`BZdC-`Yg_u)d)04hTH5{|uEFjKXx!T+h{Ixzg3ZTf6At=%bHZGvZUsjHP zz#ti2c1kP(qP}Gq(09zQ_hkM$7(4dJo9mn;xT6zP2!03!og`rG`9$u%q8cS?7Yw$% z6SAQSE^Y42ZbKGD@A1tI-dpI50>(iQ<#7Jl8Cnd30GA0c(9KC&yX7R_v`iegkWPq0 z&sNxEPB+#2FQ;y9(_sHIx94n|pUGN_@g7?M>0Np4s5$ODA9aDTBHMXz^^xRvt?_zJ z?r=4h$b(o+`jZ{B9+64C?6G`KpcpQ&pgV0hG%63~um7f>tFS&A*WfTApkB``p)P!I zwh2qwmA~%#w%2@H!c$0_U-lYf-~EI?yIV2IEC{k3{=?jQ_5W3d+*Iz=_(6`dB?Uvz zyHf}kKEv+$4J89+Qnqm#XWLP$Ii(nD{Ff$#U%V|VK=;_OrN{6ko?p1zWYq@qLED03qmE}x{;UbQ^pq$A7W6%I|w<=ptoWDsA`(3KFsY(u2z%DXrosM1UiPe>hL zpo^-!_ZU!|KvIz--q_~r@>HRT*osC<*s#g4#zbuqPZ9vJPOP)#kyWaGRPAML+(c?? zTL57ujjZ%RBME3VSl5We`Al{?%wK^PyyP?NoiD=k48J07+O5jIJHj8vCtP&ncFv<% zZrgWFa}~eC-OjOBNDr}x9I{_mg>fz?OUYUdtICv!iUV)9ksa5zTIaI~qRGE=sp67b z#7FCZmn@B43D>+6n|v|_a%zRA(&H#ftg~z{gz7%kjmoF+faWpit7!wWkYU0KNf%YZ zI64z?Ehk0X)2X&}h1Uc>;ZvfjC9}4MMSQ99(L$~kPwRDM-<-OUGvr$`TYH18^Nd!{ z@U<$&1ZB9SKSA*32|`}8LlU{cG{}2XOFtk4rA?> z0V7)U@+Gp$vRvYu$?QaXn-uOdKZ&!(R+)^nDie&iR()cNA!L3*Z|!JlZJUt6*p0`+ z3ALv6@HmgO96}!ur6UjoltbterWm)f3>iCobd&P^Dx=4Ry4Eu0w1v>G+VVM{Q>ml}Kr1&?SjH#LaCnEgC;bS2QhQ$2zGkkG zUZ!cTX{x8NYP$RwlOh`_sTYr%KFh{Tn1@+M!&#r`)mXkNcaPyfZCr3gd|(q%&|u?hMx`PM0sliu?S_{Va$jR z%~|gdzr7lQP$HdgV7Kb>Sx?vt9o5!J8_FcTeq!YIeNlm#z^xJ*kjPcW1vhKU7u7~r zzKNO<#N^XgE2Wa+biM012dgFnvn%<-WXpchmH~^x<>My2_e}`PQR7<<#}j2_Nv{ud zm<&>Bm5tG-`jR`6`Ex!8=jo>CJQ<8uaEPSq_Ug+ewSIG#yqD&_C%TdxPqf5$qKbR# zW))CbG!maz9mDf!cy`Sc{JqU{y_V~99;cyQf%R@HxUzqRjtKrnyeYcs+_Tx%rI_rN z?*6ih<)CK|ePW>ZNES`QBsfLQbkIUn*}i^jD7w!Nnn*C-2$14y}IJB`tjNy9b4!@L)P!HHls!DznOe%?lM3hl=5pM?0?4hk1W~ zu&s@3!QlMol$T2{jlD!GIkK18Hnb-xro@4g*BFzf9v@pBCDy@i4aoKepNmL+R}bwY z%Br9m>J}v~hKolMvg3R9qp~{8Pve!ZTwqr2#+4D2!?5Y)3bnRn_SV;}fTPbqckf}0 zT=%1X3Gp5W+n47i^6`$!-Hv$19?iWr4R&L*%YnjN1v@gC*`BZ4333dShArnqf6VA( z2ODZ%D?e5?G!yh!rRbwFje* z$uzIgkwtpy%E%(z#=}!%$wFH*`GJ$`AUfV&ChWtKu+MF!Z<-IcmTE_+c}3=*&!@saTt_MFG-eUoiUheG9F7We?VBMQ<} z-hKEIaNx)PRh9Es_ZdYwedv}`FYc<^yeQz{){jt>`hSf90vyTEF&^yZ3H-+Qp7CbL zvf8#~`uy-Qllsvh)o1gUe><7oKW2Q^cLaSil>j`j$m*NdFpaAqLC!Ml{LlaRjW=xt z;B%GabxS%TZ$?f&%2H7L9ZD}UcU+wUdI4dR&(jNl@wtm=0gA~XN{baWR@UFGq;A=k zSWKt*j1c$RyM2ED7>`I0&Sr0)qz!Y_Z8v6;*7d?Hz$2O6g8LyCFe#A$V)koEroEeH zZvilB1HrZeuXt`BZxF}@nLs|eX7Z!;T*5w)>! zIm{p2b_5N!2DVTVe+g;^yWY7GCg20D&g$MOR*DcEftg>u!&;@y;GJ;_-~_R*B&WJn z?&NS?Vqy3}Xx9GN!(58s@tr<^Y8%)h%zY*(^`(S0;l$T<&WAh++21Z`0(}4a} zXkzvih*04y?J+KMJkA4@A$mZKRYXmx; zV`d}qieoPfmePb=r{}TZDW6ulxZZM6^z6xGQBWV+lyg2k9B35UvLc4LrH;ig*b76X z{fDYSFsBC$cB->eoIZqs{Q(uX0`LrZMGtUh$^~c#ws9&2dm1F2dq_S{+jRDUz7RLu s@&`)&Hy$?zZu>uSyNlixoQ #include #include -#include -#include #include "gro_tcp4.h" @@ -72,20 +70,20 @@ gro_tcp4_tbl_create(uint16_t socket_id, } tbl->max_item_num = entries_num; - size = sizeof(struct gro_tcp4_key) * entries_num; - tbl->keys = rte_zmalloc_socket(__func__, + size = sizeof(struct gro_tcp4_flow) * entries_num; + tbl->flows = rte_zmalloc_socket(__func__, size, RTE_CACHE_LINE_SIZE, socket_id); - if (tbl->keys == NULL) { + if (tbl->flows == NULL) { rte_free(tbl->items); rte_free(tbl); return NULL; } - /* INVALID_ARRAY_INDEX indicates empty key */ + /* INVALID_ARRAY_INDEX indicates an empty flow */ for (i = 0; i < entries_num; i++) - tbl->keys[i].start_index = INVALID_ARRAY_INDEX; - tbl->max_key_num = entries_num; + tbl->flows[i].start_index = INVALID_ARRAY_INDEX; + tbl->max_flow_num = entries_num; return tbl; } @@ -97,111 +95,11 @@ gro_tcp4_tbl_destroy(void *tbl) if (tcp_tbl) { rte_free(tcp_tbl->items); - rte_free(tcp_tbl->keys); + rte_free(tcp_tbl->flows); } rte_free(tcp_tbl); } -/* - * merge two TCP/IPv4 packets without updating checksums. - * If cmp is larger than 0, append the new packet to the - * original packet. Otherwise, pre-pend the new packet to - * the original packet. - */ -static inline int -merge_two_tcp4_packets(struct gro_tcp4_item *item_src, - struct rte_mbuf *pkt, - uint16_t ip_id, - uint32_t sent_seq, - int cmp) -{ - struct rte_mbuf *pkt_head, *pkt_tail, *lastseg; - uint16_t tcp_datalen; - - if (cmp > 0) { - pkt_head = item_src->firstseg; - pkt_tail = pkt; - } else { - pkt_head = pkt; - pkt_tail = item_src->firstseg; - } - - /* check if the packet length will be beyond the max value */ - tcp_datalen = pkt_tail->pkt_len - pkt_tail->l2_len - - pkt_tail->l3_len - pkt_tail->l4_len; - if (pkt_head->pkt_len - pkt_head->l2_len + tcp_datalen > - TCP4_MAX_L3_LENGTH) - return 0; - - /* remove packet header for the tail packet */ - rte_pktmbuf_adj(pkt_tail, - pkt_tail->l2_len + - pkt_tail->l3_len + - pkt_tail->l4_len); - - /* chain two packets together */ - if (cmp > 0) { - item_src->lastseg->next = pkt; - item_src->lastseg = rte_pktmbuf_lastseg(pkt); - /* update IP ID to the larger value */ - item_src->ip_id = ip_id; - } else { - lastseg = rte_pktmbuf_lastseg(pkt); - lastseg->next = item_src->firstseg; - item_src->firstseg = pkt; - /* update sent_seq to the smaller value */ - item_src->sent_seq = sent_seq; - } - item_src->nb_merged++; - - /* update mbuf metadata for the merged packet */ - pkt_head->nb_segs += pkt_tail->nb_segs; - pkt_head->pkt_len += pkt_tail->pkt_len; - - return 1; -} - -static inline int -check_seq_option(struct gro_tcp4_item *item, - struct tcp_hdr *tcp_hdr, - uint16_t tcp_hl, - uint16_t tcp_dl, - uint16_t ip_id, - uint32_t sent_seq) -{ - struct rte_mbuf *pkt0 = item->firstseg; - struct ipv4_hdr *ipv4_hdr0; - struct tcp_hdr *tcp_hdr0; - uint16_t tcp_hl0, tcp_dl0; - uint16_t len; - - ipv4_hdr0 = (struct ipv4_hdr *)(rte_pktmbuf_mtod(pkt0, char *) + - pkt0->l2_len); - tcp_hdr0 = (struct tcp_hdr *)((char *)ipv4_hdr0 + pkt0->l3_len); - tcp_hl0 = pkt0->l4_len; - - /* check if TCP option fields equal. If not, return 0. */ - len = RTE_MAX(tcp_hl, tcp_hl0) - sizeof(struct tcp_hdr); - if ((tcp_hl != tcp_hl0) || - ((len > 0) && (memcmp(tcp_hdr + 1, - tcp_hdr0 + 1, - len) != 0))) - return 0; - - /* check if the two packets are neighbors */ - tcp_dl0 = pkt0->pkt_len - pkt0->l2_len - pkt0->l3_len - tcp_hl0; - if ((sent_seq == (item->sent_seq + tcp_dl0)) && - (ip_id == (item->ip_id + 1))) - /* append the new packet */ - return 1; - else if (((sent_seq + tcp_dl) == item->sent_seq) && - ((ip_id + item->nb_merged) == item->ip_id)) - /* pre-pend the new packet */ - return -1; - else - return 0; -} - static inline uint32_t find_an_empty_item(struct gro_tcp4_tbl *tbl) { @@ -215,13 +113,13 @@ find_an_empty_item(struct gro_tcp4_tbl *tbl) } static inline uint32_t -find_an_empty_key(struct gro_tcp4_tbl *tbl) +find_an_empty_flow(struct gro_tcp4_tbl *tbl) { uint32_t i; - uint32_t max_key_num = tbl->max_key_num; + uint32_t max_flow_num = tbl->max_flow_num; - for (i = 0; i < max_key_num; i++) - if (tbl->keys[i].start_index == INVALID_ARRAY_INDEX) + for (i = 0; i < max_flow_num; i++) + if (tbl->flows[i].start_index == INVALID_ARRAY_INDEX) return i; return INVALID_ARRAY_INDEX; } @@ -229,10 +127,11 @@ find_an_empty_key(struct gro_tcp4_tbl *tbl) static inline uint32_t insert_new_item(struct gro_tcp4_tbl *tbl, struct rte_mbuf *pkt, - uint16_t ip_id, - uint32_t sent_seq, + uint64_t start_time, uint32_t prev_idx, - uint64_t start_time) + uint32_t sent_seq, + uint16_t ip_id, + uint8_t is_atomic) { uint32_t item_idx; @@ -247,9 +146,10 @@ insert_new_item(struct gro_tcp4_tbl *tbl, tbl->items[item_idx].sent_seq = sent_seq; tbl->items[item_idx].ip_id = ip_id; tbl->items[item_idx].nb_merged = 1; + tbl->items[item_idx].is_atomic = is_atomic; tbl->item_num++; - /* if the previous packet exists, chain the new one with it */ + /* If the previous packet exists, chain them together. */ if (prev_idx != INVALID_ARRAY_INDEX) { tbl->items[item_idx].next_pkt_idx = tbl->items[prev_idx].next_pkt_idx; @@ -260,12 +160,13 @@ insert_new_item(struct gro_tcp4_tbl *tbl, } static inline uint32_t -delete_item(struct gro_tcp4_tbl *tbl, uint32_t item_idx, +delete_item(struct gro_tcp4_tbl *tbl, + uint32_t item_idx, uint32_t prev_item_idx) { uint32_t next_idx = tbl->items[item_idx].next_pkt_idx; - /* set NULL to firstseg to indicate it's an empty item */ + /* NULL indicates an empty item. */ tbl->items[item_idx].firstseg = NULL; tbl->item_num--; if (prev_item_idx != INVALID_ARRAY_INDEX) @@ -275,53 +176,33 @@ delete_item(struct gro_tcp4_tbl *tbl, uint32_t item_idx, } static inline uint32_t -insert_new_key(struct gro_tcp4_tbl *tbl, - struct tcp4_key *key_src, +insert_new_flow(struct gro_tcp4_tbl *tbl, + struct tcp4_flow_key *src, uint32_t item_idx) { - struct tcp4_key *key_dst; - uint32_t key_idx; + struct tcp4_flow_key *dst; + uint32_t flow_idx; - key_idx = find_an_empty_key(tbl); - if (key_idx == INVALID_ARRAY_INDEX) + flow_idx = find_an_empty_flow(tbl); + if (flow_idx == INVALID_ARRAY_INDEX) return INVALID_ARRAY_INDEX; - key_dst = &(tbl->keys[key_idx].key); + dst = &(tbl->flows[flow_idx].key); - ether_addr_copy(&(key_src->eth_saddr), &(key_dst->eth_saddr)); - ether_addr_copy(&(key_src->eth_daddr), &(key_dst->eth_daddr)); - key_dst->ip_src_addr = key_src->ip_src_addr; - key_dst->ip_dst_addr = key_src->ip_dst_addr; - key_dst->recv_ack = key_src->recv_ack; - key_dst->src_port = key_src->src_port; - key_dst->dst_port = key_src->dst_port; + ether_addr_copy(&(src->eth_saddr), &(dst->eth_saddr)); + ether_addr_copy(&(src->eth_daddr), &(dst->eth_daddr)); + dst->ip_src_addr = src->ip_src_addr; + dst->ip_dst_addr = src->ip_dst_addr; + dst->recv_ack = src->recv_ack; + dst->src_port = src->src_port; + dst->dst_port = src->dst_port; - /* non-INVALID_ARRAY_INDEX value indicates this key is valid */ - tbl->keys[key_idx].start_index = item_idx; - tbl->key_num++; + tbl->flows[flow_idx].start_index = item_idx; + tbl->flow_num++; - return key_idx; + return flow_idx; } -static inline int -is_same_key(struct tcp4_key k1, struct tcp4_key k2) -{ - if (is_same_ether_addr(&k1.eth_saddr, &k2.eth_saddr) == 0) - return 0; - - if (is_same_ether_addr(&k1.eth_daddr, &k2.eth_daddr) == 0) - return 0; - - return ((k1.ip_src_addr == k2.ip_src_addr) && - (k1.ip_dst_addr == k2.ip_dst_addr) && - (k1.recv_ack == k2.recv_ack) && - (k1.src_port == k2.src_port) && - (k1.dst_port == k2.dst_port)); -} - -/* - * update packet length for the flushed packet. - */ static inline void update_header(struct gro_tcp4_item *item) { @@ -343,30 +224,40 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, struct ipv4_hdr *ipv4_hdr; struct tcp_hdr *tcp_hdr; uint32_t sent_seq; - uint16_t tcp_dl, ip_id; + uint16_t ip_id, frag_off, tcp_dl, hdr_len; + uint8_t is_atomic; - struct tcp4_key key; + struct tcp4_flow_key key; uint32_t cur_idx, prev_idx, item_idx; - uint32_t i, max_key_num; + uint32_t i, max_flow_num; int cmp; eth_hdr = rte_pktmbuf_mtod(pkt, struct ether_hdr *); ipv4_hdr = (struct ipv4_hdr *)((char *)eth_hdr + pkt->l2_len); tcp_hdr = (struct tcp_hdr *)((char *)ipv4_hdr + pkt->l3_len); + hdr_len = pkt->l2_len + pkt->l3_len + pkt->l4_len; /* - * if FIN, SYN, RST, PSH, URG, ECE or - * CWR is set, return immediately. + * Don't process the packet which has FIN, SYN, RST, PSH, URG, ECE + * or CWR set. */ if (tcp_hdr->tcp_flags != TCP_ACK_FLAG) return -1; - /* if payload length is 0, return immediately */ - tcp_dl = rte_be_to_cpu_16(ipv4_hdr->total_length) - pkt->l3_len - - pkt->l4_len; - if (tcp_dl == 0) + /* + * Don't process the packet whose payload length is less than or + * equal to 0. + */ + tcp_dl = pkt->pkt_len - hdr_len; + if (tcp_dl <= 0) return -1; - ip_id = rte_be_to_cpu_16(ipv4_hdr->packet_id); + /* + * Save IP ID for the packet whose DF bit is 0. For the packet + * whose DF bit is 1, IP ID is ignored. + */ + frag_off = rte_be_to_cpu_16(ipv4_hdr->fragment_offset); + is_atomic = (frag_off & IPV4_HDR_DF_FLAG) == IPV4_HDR_DF_FLAG; + ip_id = is_atomic ? 0 : rte_be_to_cpu_16(ipv4_hdr->packet_id); sent_seq = rte_be_to_cpu_32(tcp_hdr->sent_seq); ether_addr_copy(&(eth_hdr->s_addr), &(key.eth_saddr)); @@ -377,50 +268,55 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, key.dst_port = tcp_hdr->dst_port; key.recv_ack = tcp_hdr->recv_ack; - /* search for a key */ - max_key_num = tbl->max_key_num; - for (i = 0; i < max_key_num; i++) { - if ((tbl->keys[i].start_index != INVALID_ARRAY_INDEX) && - is_same_key(tbl->keys[i].key, key)) + /* Search for a matched flow. */ + max_flow_num = tbl->max_flow_num; + for (i = 0; i < max_flow_num; i++) { + if ((tbl->flows[i].start_index != INVALID_ARRAY_INDEX) && + is_same_tcp4_flow(tbl->flows[i].key, key)) break; } - /* can't find a key, so insert a new key and a new item. */ - if (i == tbl->max_key_num) { - item_idx = insert_new_item(tbl, pkt, ip_id, sent_seq, - INVALID_ARRAY_INDEX, start_time); + /* + * Fail to find a matched flow. Insert a new flow and store the + * packet into the flow. + */ + if (i == tbl->max_flow_num) { + item_idx = insert_new_item(tbl, pkt, start_time, + INVALID_ARRAY_INDEX, sent_seq, ip_id, + is_atomic); if (item_idx == INVALID_ARRAY_INDEX) return -1; - if (insert_new_key(tbl, &key, item_idx) == + if (insert_new_flow(tbl, &key, item_idx) == INVALID_ARRAY_INDEX) { - /* - * fail to insert a new key, so - * delete the inserted item - */ + /* Fail to insert a new flow. */ delete_item(tbl, item_idx, INVALID_ARRAY_INDEX); return -1; } return 0; } - /* traverse all packets in the item group to find one to merge */ - cur_idx = tbl->keys[i].start_index; + /* + * Check all packets in the flow and try to find a neighbor for + * the input packet. + */ + cur_idx = tbl->flows[i].start_index; prev_idx = cur_idx; do { cmp = check_seq_option(&(tbl->items[cur_idx]), tcp_hdr, - pkt->l4_len, tcp_dl, ip_id, sent_seq); + sent_seq, ip_id, pkt->l4_len, tcp_dl, 0, + is_atomic); if (cmp) { if (merge_two_tcp4_packets(&(tbl->items[cur_idx]), - pkt, ip_id, - sent_seq, cmp)) + pkt, cmp, sent_seq, ip_id, 0)) return 1; /* - * fail to merge two packets since the packet - * length will be greater than the max value. - * So insert the packet into the item group. + * Fail to merge the two packets, as the packet + * length is greater than the max value. Store + * the packet into the flow. */ - if (insert_new_item(tbl, pkt, ip_id, sent_seq, - prev_idx, start_time) == + if (insert_new_item(tbl, pkt, start_time, prev_idx, + sent_seq, ip_id, + is_atomic) == INVALID_ARRAY_INDEX) return -1; return 0; @@ -429,12 +325,9 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, cur_idx = tbl->items[cur_idx].next_pkt_idx; } while (cur_idx != INVALID_ARRAY_INDEX); - /* - * can't find a packet in the item group to merge, - * so insert the packet into the item group. - */ - if (insert_new_item(tbl, pkt, ip_id, sent_seq, prev_idx, - start_time) == INVALID_ARRAY_INDEX) + /* Fail to find a neighbor, so store the packet into the flow. */ + if (insert_new_item(tbl, pkt, start_time, prev_idx, sent_seq, + ip_id, is_atomic) == INVALID_ARRAY_INDEX) return -1; return 0; @@ -446,46 +339,35 @@ gro_tcp4_tbl_timeout_flush(struct gro_tcp4_tbl *tbl, struct rte_mbuf **out, uint16_t nb_out) { - uint16_t k = 0; + uint32_t max_flow_num = tbl->max_flow_num; uint32_t i, j; - uint32_t max_key_num = tbl->max_key_num; + uint16_t k = 0; - for (i = 0; i < max_key_num; i++) { - /* all keys have been checked, return immediately */ - if (tbl->key_num == 0) + for (i = 0; i < max_flow_num; i++) { + if (unlikely(tbl->flow_num == 0)) return k; - j = tbl->keys[i].start_index; + j = tbl->flows[i].start_index; while (j != INVALID_ARRAY_INDEX) { if (tbl->items[j].start_time <= flush_timestamp) { out[k++] = tbl->items[j].firstseg; if (tbl->items[j].nb_merged > 1) update_header(&(tbl->items[j])); /* - * delete the item and get - * the next packet index + * Delete the packet and get the next + * packet in the flow. */ - j = delete_item(tbl, j, - INVALID_ARRAY_INDEX); + j = delete_item(tbl, j, INVALID_ARRAY_INDEX); + tbl->flows[i].start_index = j; + if (j == INVALID_ARRAY_INDEX) + tbl->flow_num--; - /* - * delete the key as all of - * packets are flushed - */ - if (j == INVALID_ARRAY_INDEX) { - tbl->keys[i].start_index = - INVALID_ARRAY_INDEX; - tbl->key_num--; - } else - /* update start_index of the key */ - tbl->keys[i].start_index = j; - - if (k == nb_out) + if (unlikely(k == nb_out)) return k; } else /* - * left packets of this key won't be - * timeout, so go to check other keys. + * The left packets in this flow won't be + * timeout. Go to check other flows. */ break; } diff --git a/lib/librte_gro/gro_tcp4.h b/lib/librte_gro/gro_tcp4.h index 0a81716..9ac8bc9 100644 --- a/lib/librte_gro/gro_tcp4.h +++ b/lib/librte_gro/gro_tcp4.h @@ -33,17 +33,20 @@ #ifndef _GRO_TCP4_H_ #define _GRO_TCP4_H_ +#include +#include + #define INVALID_ARRAY_INDEX 0xffffffffUL #define GRO_TCP4_TBL_MAX_ITEM_NUM (1024UL * 1024UL) /* - * the max L3 length of a TCP/IPv4 packet. The L3 length - * is the sum of ipv4 header, tcp header and L4 payload. + * The max length of a IPv4 packet, which includes the length of the L3 + * header, the L4 header and the data payload. */ -#define TCP4_MAX_L3_LENGTH UINT16_MAX +#define MAX_IPV4_PKT_LENGTH UINT16_MAX -/* criteria of mergeing packets */ -struct tcp4_key { +/* Header fields representing a TCP/IPv4 flow */ +struct tcp4_flow_key { struct ether_addr eth_saddr; struct ether_addr eth_daddr; uint32_t ip_src_addr; @@ -54,77 +57,76 @@ struct tcp4_key { uint16_t dst_port; }; -struct gro_tcp4_key { - struct tcp4_key key; +struct gro_tcp4_flow { + struct tcp4_flow_key key; /* - * the index of the first packet in the item group. - * If the value is INVALID_ARRAY_INDEX, it means - * the key is empty. + * The index of the first packet in the flow. + * INVALID_ARRAY_INDEX indicates an empty flow. */ uint32_t start_index; }; struct gro_tcp4_item { /* - * first segment of the packet. If the value + * The first MBUF segment of the packet. If the value * is NULL, it means the item is empty. */ struct rte_mbuf *firstseg; - /* last segment of the packet */ + /* The last MBUF segment of the packet */ struct rte_mbuf *lastseg; /* - * the time when the first packet is inserted - * into the table. If a packet in the table is - * merged with an incoming packet, this value - * won't be updated. We set this value only - * when the first packet is inserted into the - * table. + * The time when the first packet is inserted into the table. + * This value won't be updated, even if the packet is merged + * with other packets. */ uint64_t start_time; /* - * we use next_pkt_idx to chain the packets that - * have same key value but can't be merged together. + * next_pkt_idx is used to chain the packets that + * are in the same flow but can't be merged together + * (e.g. caused by packet reordering). */ uint32_t next_pkt_idx; - /* the sequence number of the packet */ + /* TCP sequence number of the packet */ uint32_t sent_seq; - /* the IP ID of the packet */ + /* IPv4 ID of the packet */ uint16_t ip_id; - /* the number of merged packets */ + /* The number of merged packets */ uint16_t nb_merged; + /* Indicate if IP ID can be ignored */ + uint8_t is_atomic; }; /* - * TCP/IPv4 reassembly table structure. + * TCP/IPv4 reassembly table structure */ struct gro_tcp4_tbl { /* item array */ struct gro_tcp4_item *items; - /* key array */ - struct gro_tcp4_key *keys; + /* flow array */ + struct gro_tcp4_flow *flows; /* current item number */ uint32_t item_num; - /* current key num */ - uint32_t key_num; + /* current flow num */ + uint32_t flow_num; /* item array size */ uint32_t max_item_num; - /* key array size */ - uint32_t max_key_num; + /* flow array size */ + uint32_t max_flow_num; }; /** * This function creates a TCP/IPv4 reassembly table. * * @param socket_id - * socket index for allocating TCP/IPv4 reassemble table + * Socket index for allocating the TCP/IPv4 reassemble table * @param max_flow_num - * the maximum number of flows in the TCP/IPv4 GRO table + * The maximum number of flows in the TCP/IPv4 GRO table * @param max_item_per_flow - * the maximum packet number per flow. + * The maximum number of packets per flow * * @return - * if create successfully, return a pointer which points to the - * created TCP/IPv4 GRO table. Otherwise, return NULL. + * - Return the table pointer on success. + * - Return NULL on failure. */ void *gro_tcp4_tbl_create(uint16_t socket_id, uint16_t max_flow_num, @@ -134,62 +136,55 @@ void *gro_tcp4_tbl_create(uint16_t socket_id, * This function destroys a TCP/IPv4 reassembly table. * * @param tbl - * a pointer points to the TCP/IPv4 reassembly table. + * Pointer pointint to the TCP/IPv4 reassembly table. */ void gro_tcp4_tbl_destroy(void *tbl); /** - * This function searches for a packet in the TCP/IPv4 reassembly table - * to merge with the inputted one. To merge two packets is to chain them - * together and update packet headers. Packets, whose SYN, FIN, RST, PSH - * CWR, ECE or URG bit is set, are returned immediately. Packets which - * only have packet headers (i.e. without data) are also returned - * immediately. Otherwise, the packet is either merged, or inserted into - * the table. Besides, if there is no available space to insert the - * packet, this function returns immediately too. + * This function merges a TCP/IPv4 packet. It doesn't process the packet, + * which has SYN, FIN, RST, PSH, CWR, ECE or URG set, or doesn't have + * payload. The packet is returned if it has invalid parameters or there + * is no available space in the table. * - * This function assumes the inputted packet is with correct IPv4 and - * TCP checksums. And if two packets are merged, it won't re-calculate - * IPv4 and TCP checksums. Besides, if the inputted packet is IP - * fragmented, it assumes the packet is complete (with TCP header). + * This function doesn't check if the packet has correct checksums and + * doesn't re-calculate checksums for the merged packet. Additionally, + * it assumes the packets are complete (i.e., MF==0 && frag_off==0), + * when IP fragmentation is possible (i.e., DF==0). * * @param pkt - * packet to reassemble. + * Packet to reassemble * @param tbl - * a pointer that points to a TCP/IPv4 reassembly table. + * Pointer pointing to the TCP/IPv4 reassembly table * @start_time - * the start time that the packet is inserted into the table + * The time when the packet is inserted into the table * * @return - * if the packet doesn't have data, or SYN, FIN, RST, PSH, CWR, ECE - * or URG bit is set, or there is no available space in the table to - * insert a new item or a new key, return a negative value. If the - * packet is merged successfully, return an positive value. If the - * packet is inserted into the table, return 0. + * - Return a positive value if the input packet is merged. + * - Return zero if the input packet isn't merged but stored in the table. + * - Return a negative value for invalid parameters or no available + * space in the table. */ int32_t gro_tcp4_reassemble(struct rte_mbuf *pkt, struct gro_tcp4_tbl *tbl, uint64_t start_time); /** - * This function flushes timeout packets in a TCP/IPv4 reassembly table - * to applications, and without updating checksums for merged packets. - * The max number of flushed timeout packets is the element number of - * the array which is used to keep flushed packets. + * This function flushes timeout packets in a TCP/IPv4 reassembly table, + * and without updating checksums. * * @param tbl - * a pointer that points to a TCP GRO table. + * TCP/IPv4 reassembly table pointer * @param flush_timestamp - * this function flushes packets which are inserted into the table - * before or at the flush_timestamp. + * Flush packets which are inserted into the table before or at the + * flush_timestamp. * @param out - * pointer array which is used to keep flushed packets. + * Pointer array used to keep flushed packets * @param nb_out - * the element number of out. It's also the max number of timeout + * The element number in 'out'. It also determines the maximum number of * packets that can be flushed finally. * * @return - * the number of packets that are returned. + * The number of flushed packets */ uint16_t gro_tcp4_tbl_timeout_flush(struct gro_tcp4_tbl *tbl, uint64_t flush_timestamp, @@ -201,10 +196,130 @@ uint16_t gro_tcp4_tbl_timeout_flush(struct gro_tcp4_tbl *tbl, * reassembly table. * * @param tbl - * pointer points to a TCP/IPv4 reassembly table. + * TCP/IPv4 reassembly table pointer * * @return - * the number of packets in the table + * The number of packets in the table */ uint32_t gro_tcp4_tbl_pkt_count(void *tbl); + +/* + * Check if two TCP/IPv4 packets belong to the same flow. + */ +static inline int +is_same_tcp4_flow(struct tcp4_flow_key k1, struct tcp4_flow_key k2) +{ + return (is_same_ether_addr(&k1.eth_saddr, &k2.eth_saddr) && + is_same_ether_addr(&k1.eth_daddr, &k2.eth_daddr) && + (k1.ip_src_addr == k2.ip_src_addr) && + (k1.ip_dst_addr == k2.ip_dst_addr) && + (k1.recv_ack == k2.recv_ack) && + (k1.src_port == k2.src_port) && + (k1.dst_port == k2.dst_port)); +} + +/* + * Check if two TCP/IPv4 packets are neighbors. + */ +static inline int +check_seq_option(struct gro_tcp4_item *item, + struct tcp_hdr *tcph, + uint32_t sent_seq, + uint16_t ip_id, + uint16_t tcp_hl, + uint16_t tcp_dl, + uint16_t l2_offset, + uint8_t is_atomic) +{ + struct rte_mbuf *pkt_orig = item->firstseg; + struct ipv4_hdr *iph_orig; + struct tcp_hdr *tcph_orig; + uint16_t len, l4_len_orig; + + iph_orig = (struct ipv4_hdr *)(rte_pktmbuf_mtod(pkt_orig, char *) + + l2_offset + pkt_orig->l2_len); + tcph_orig = (struct tcp_hdr *)((char *)iph_orig + pkt_orig->l3_len); + l4_len_orig = pkt_orig->l4_len; + + /* Check if TCP option fields equal */ + len = RTE_MAX(tcp_hl, l4_len_orig) - sizeof(struct tcp_hdr); + if ((tcp_hl != l4_len_orig) || ((len > 0) && + (memcmp(tcph + 1, tcph_orig + 1, + len) != 0))) + return 0; + + /* Don't merge packets whose DF bits are different */ + if (item->is_atomic ^ is_atomic) + return 0; + + /* Check if the two packets are neighbors */ + len = pkt_orig->pkt_len - l2_offset - pkt_orig->l2_len - + pkt_orig->l3_len - l4_len_orig; + if ((sent_seq == item->sent_seq + len) && (is_atomic || + (ip_id == item->ip_id + item->nb_merged))) + /* Append the new packet */ + return 1; + else if ((sent_seq + tcp_dl == item->sent_seq) && (is_atomic || + (ip_id + 1 == item->ip_id))) + /* Pre-pend the new packet */ + return -1; + else + return 0; +} + +/* + * Merge two TCP/IPv4 packets without updating checksums. + * If cmp is larger than 0, append the new packet to the + * original packet. Otherwise, pre-pend the new packet to + * the original packet. + */ +static inline int +merge_two_tcp4_packets(struct gro_tcp4_item *item, + struct rte_mbuf *pkt, + int cmp, + uint32_t sent_seq, + uint16_t ip_id, + uint16_t l2_offset) +{ + struct rte_mbuf *pkt_head, *pkt_tail, *lastseg; + uint16_t hdr_len; + + if (cmp > 0) { + pkt_head = item->firstseg; + pkt_tail = pkt; + } else { + pkt_head = pkt; + pkt_tail = item->firstseg; + } + + /* Check if the length is greater than the max value */ + hdr_len = l2_offset + pkt_head->l2_len + pkt_head->l3_len + + pkt_head->l4_len; + if (pkt_head->pkt_len - l2_offset - pkt_head->l2_len + + pkt_tail->pkt_len - hdr_len > MAX_IPV4_PKT_LENGTH) + return 0; + + /* Remove the packet header */ + rte_pktmbuf_adj(pkt_tail, hdr_len); + + /* Chain two packets together */ + if (cmp > 0) { + item->lastseg->next = pkt; + item->lastseg = rte_pktmbuf_lastseg(pkt); + } else { + lastseg = rte_pktmbuf_lastseg(pkt); + lastseg->next = item->firstseg; + item->firstseg = pkt; + /* Update sent_seq and ip_id */ + item->sent_seq = sent_seq; + item->ip_id = ip_id; + } + item->nb_merged++; + + /* Update MBUF metadata for the merged packet */ + pkt_head->nb_segs += pkt_tail->nb_segs; + pkt_head->pkt_len += pkt_tail->pkt_len; + + return 1; +} #endif diff --git a/lib/librte_gro/rte_gro.c b/lib/librte_gro/rte_gro.c index 7853246..d43f8e8 100644 --- a/lib/librte_gro/rte_gro.c +++ b/lib/librte_gro/rte_gro.c @@ -51,11 +51,14 @@ static gro_tbl_destroy_fn tbl_destroy_fn[RTE_GRO_TYPE_MAX_NUM] = { static gro_tbl_pkt_count_fn tbl_pkt_count_fn[RTE_GRO_TYPE_MAX_NUM] = { gro_tcp4_tbl_pkt_count, NULL}; +#define IS_IPV4_TCP_PKT(ptype) (RTE_ETH_IS_IPV4_HDR(ptype) && \ + ((ptype & RTE_PTYPE_L4_TCP) == RTE_PTYPE_L4_TCP)) + /* - * GRO context structure, which is used to merge packets. It keeps - * many reassembly tables of desired GRO types. Applications need to - * create GRO context objects before using rte_gro_reassemble to - * perform GRO. + * GRO context structure. It keeps the table structures, which are + * used to merge packets, for different GRO types. Before using + * rte_gro_reassemble(), applications need to create the GRO context + * first. */ struct gro_ctx { /* GRO types to perform */ @@ -93,7 +96,7 @@ rte_gro_ctx_create(const struct rte_gro_param *param) param->max_flow_num, param->max_item_per_flow); if (gro_ctx->tbls[i] == NULL) { - /* destroy all created tables */ + /* Destroy all created tables */ gro_ctx->gro_types = gro_types; rte_gro_ctx_destroy(gro_ctx); return NULL; @@ -131,62 +134,55 @@ rte_gro_reassemble_burst(struct rte_mbuf **pkts, uint16_t nb_pkts, const struct rte_gro_param *param) { - uint16_t i; - uint16_t nb_after_gro = nb_pkts; - uint32_t item_num; - - /* allocate a reassembly table for TCP/IPv4 GRO */ + /* Allocate a reassembly table for TCP/IPv4 GRO */ struct gro_tcp4_tbl tcp_tbl; - struct gro_tcp4_key tcp_keys[RTE_GRO_MAX_BURST_ITEM_NUM]; + struct gro_tcp4_flow tcp_flows[RTE_GRO_MAX_BURST_ITEM_NUM]; struct gro_tcp4_item tcp_items[RTE_GRO_MAX_BURST_ITEM_NUM] = {{0} }; struct rte_mbuf *unprocess_pkts[nb_pkts]; - uint16_t unprocess_num = 0; - int32_t ret; uint64_t current_time; + uint32_t item_num; + int32_t ret; + uint16_t i, unprocess_num = 0, nb_after_gro = nb_pkts; if ((param->gro_types & RTE_GRO_TCP_IPV4) == 0) return nb_pkts; - /* get the actual number of packets */ + /* Get the actual number of packets */ item_num = RTE_MIN(nb_pkts, (param->max_flow_num * - param->max_item_per_flow)); + param->max_item_per_flow)); item_num = RTE_MIN(item_num, RTE_GRO_MAX_BURST_ITEM_NUM); for (i = 0; i < item_num; i++) - tcp_keys[i].start_index = INVALID_ARRAY_INDEX; + tcp_flows[i].start_index = INVALID_ARRAY_INDEX; - tcp_tbl.keys = tcp_keys; + tcp_tbl.flows = tcp_flows; tcp_tbl.items = tcp_items; - tcp_tbl.key_num = 0; + tcp_tbl.flow_num = 0; tcp_tbl.item_num = 0; - tcp_tbl.max_key_num = item_num; + tcp_tbl.max_flow_num = item_num; tcp_tbl.max_item_num = item_num; current_time = rte_rdtsc(); for (i = 0; i < nb_pkts; i++) { - if ((pkts[i]->packet_type & (RTE_PTYPE_L3_IPV4 | - RTE_PTYPE_L4_TCP)) == - (RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_TCP)) { - ret = gro_tcp4_reassemble(pkts[i], - &tcp_tbl, + if (IS_IPV4_TCP_PKT(pkts[i]->packet_type)) { + ret = gro_tcp4_reassemble(pkts[i], &tcp_tbl, current_time); if (ret > 0) - /* merge successfully */ + /* Merge successfully */ nb_after_gro--; - else if (ret < 0) { - unprocess_pkts[unprocess_num++] = - pkts[i]; - } + else if (ret < 0) + unprocess_pkts[unprocess_num++] = pkts[i]; } else unprocess_pkts[unprocess_num++] = pkts[i]; } - /* re-arrange GROed packets */ if (nb_after_gro < nb_pkts) { + /* Flush packets from the tables */ i = gro_tcp4_tbl_timeout_flush(&tcp_tbl, current_time, pkts, nb_pkts); + /* Copy unprocessed packets */ if (unprocess_num > 0) { memcpy(&pkts[i], unprocess_pkts, sizeof(struct rte_mbuf *) * @@ -202,31 +198,28 @@ rte_gro_reassemble(struct rte_mbuf **pkts, uint16_t nb_pkts, void *ctx) { - uint16_t i, unprocess_num = 0; struct rte_mbuf *unprocess_pkts[nb_pkts]; struct gro_ctx *gro_ctx = ctx; + void *tcp_tbl; uint64_t current_time; + uint16_t i, unprocess_num = 0; if ((gro_ctx->gro_types & RTE_GRO_TCP_IPV4) == 0) return nb_pkts; + tcp_tbl = gro_ctx->tbls[RTE_GRO_TCP_IPV4_INDEX]; current_time = rte_rdtsc(); for (i = 0; i < nb_pkts; i++) { - if ((pkts[i]->packet_type & (RTE_PTYPE_L3_IPV4 | - RTE_PTYPE_L4_TCP)) == - (RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_TCP)) { - if (gro_tcp4_reassemble(pkts[i], - gro_ctx->tbls - [RTE_GRO_TCP_IPV4_INDEX], + if (IS_IPV4_TCP_PKT(pkts[i]->packet_type)) { + if (gro_tcp4_reassemble(pkts[i], tcp_tbl, current_time) < 0) unprocess_pkts[unprocess_num++] = pkts[i]; } else unprocess_pkts[unprocess_num++] = pkts[i]; } if (unprocess_num > 0) { - memcpy(pkts, unprocess_pkts, - sizeof(struct rte_mbuf *) * + memcpy(pkts, unprocess_pkts, sizeof(struct rte_mbuf *) * unprocess_num); } @@ -252,6 +245,7 @@ rte_gro_timeout_flush(void *ctx, flush_timestamp, out, max_nb_out); } + return 0; } @@ -274,5 +268,6 @@ rte_gro_get_pkt_count(void *ctx) continue; item_num += pkt_count_fn(gro_ctx->tbls[i]); } + return item_num; } diff --git a/lib/librte_gro/rte_gro.h b/lib/librte_gro/rte_gro.h index d57e0c5..c28b3a2 100644 --- a/lib/librte_gro/rte_gro.h +++ b/lib/librte_gro/rte_gro.h @@ -59,8 +59,8 @@ extern "C" { /**< TCP/IPv4 GRO flag */ /** - * A structure which is used to create GRO context objects or tell - * rte_gro_reassemble_burst() what reassembly rules are demanded. + * Structure used to create GRO context objects or used to pass + * application-determined parameters to rte_gro_reassemble_burst(). */ struct rte_gro_param { uint64_t gro_types; @@ -106,26 +106,23 @@ void rte_gro_ctx_destroy(void *ctx); /** * This is one of the main reassembly APIs, which merges numbers of - * packets at a time. It assumes that all inputted packets are with - * correct checksums. That is, applications should guarantee all - * inputted packets are correct. Besides, it doesn't re-calculate - * checksums for merged packets. If inputted packets are IP fragmented, - * this function assumes them are complete (i.e. with L4 header). After - * finishing processing, it returns all GROed packets to applications - * immediately. + * packets at a time. It doesn't check if input packets have correct + * checksums and doesn't re-calculate checksums for merged packets. + * It assumes the packets are complete (i.e., MF==0 && frag_off==0), + * when IP fragmentation is possible (i.e., DF==1). The GROed packets + * are returned as soon as the function finishes. * * @param pkts - * a pointer array which points to the packets to reassemble. Besides, - * it keeps mbuf addresses for the GROed packets. + * Pointer array pointing to the packets to reassemble. Besides, it + * keeps MBUF addresses for the GROed packets. * @param nb_pkts - * the number of packets to reassemble. + * The number of packets to reassemble * @param param - * applications use it to tell rte_gro_reassemble_burst() what rules - * are demanded. + * Application-determined parameters for reassembling packets. * * @return - * the number of packets after been GROed. If no packets are merged, - * the returned value is nb_pkts. + * The number of packets after been GROed. If no packets are merged, + * the return value is equals to nb_pkts. */ uint16_t rte_gro_reassemble_burst(struct rte_mbuf **pkts, uint16_t nb_pkts, @@ -135,32 +132,28 @@ uint16_t rte_gro_reassemble_burst(struct rte_mbuf **pkts, * @warning * @b EXPERIMENTAL: this API may change without prior notice * - * Reassembly function, which tries to merge inputted packets with - * the packets in the reassembly tables of a given GRO context. This - * function assumes all inputted packets are with correct checksums. - * And it won't update checksums if two packets are merged. Besides, - * if inputted packets are IP fragmented, this function assumes they - * are complete packets (i.e. with L4 header). + * Reassembly function, which tries to merge input packets with the + * existed packets in the reassembly tables of a given GRO context. + * It doesn't check if input packets have correct checksums and doesn't + * re-calculate checksums for merged packets. Additionally, it assumes + * the packets are complete (i.e., MF==0 && frag_off==0), when IP + * fragmentation is possible (i.e., DF==1). * - * If the inputted packets don't have data or are with unsupported GRO - * types etc., they won't be processed and are returned to applications. - * Otherwise, the inputted packets are either merged or inserted into - * the table. If applications want get packets in the table, they need - * to call flush API. + * If the input packets have invalid parameters (e.g. no data payload, + * unsupported GRO types), they are returned to applications. Otherwise, + * they are either merged or inserted into the table. Applications need + * to flush packets from the tables by flush API, if they want to get the + * GROed packets. * * @param pkts - * packet to reassemble. Besides, after this function finishes, it - * keeps the unprocessed packets (e.g. without data or unsupported - * GRO types). + * Packets to reassemble. It's also used to store the unprocessed packets. * @param nb_pkts - * the number of packets to reassemble. + * The number of packets to reassemble * @param ctx - * a pointer points to a GRO context object. + * GRO context object pointer * * @return - * return the number of unprocessed packets (e.g. without data or - * unsupported GRO types). If all packets are processed (merged or - * inserted into the table), return 0. + * - The number of unprocessed packets. */ uint16_t rte_gro_reassemble(struct rte_mbuf **pkts, uint16_t nb_pkts, @@ -170,25 +163,24 @@ uint16_t rte_gro_reassemble(struct rte_mbuf **pkts, * @warning * @b EXPERIMENTAL: this API may change without prior notice * - * This function flushes the timeout packets from reassembly tables of - * desired GRO types. The max number of flushed timeout packets is the - * element number of the array which is used to keep the flushed packets. + * This function flushes the timeout packets from the reassembly tables + * of desired GRO types. The max number of flushed packets is the + * element number of 'out'. * - * Besides, this function won't re-calculate checksums for merged - * packets in the tables. That is, the returned packets may be with - * wrong checksums. + * Additionally, the flushed packets may have incorrect checksums, since + * this function doesn't re-calculate checksums for merged packets. * * @param ctx - * a pointer points to a GRO context object. + * GRO context object pointer. * @param timeout_cycles - * max TTL for packets in reassembly tables, measured in nanosecond. + * The max TTL for packets in reassembly tables, measured in nanosecond. * @param gro_types - * this function only flushes packets which belong to the GRO types - * specified by gro_types. + * This function flushes packets whose GRO types are specified by + * gro_types. * @param out - * a pointer array that is used to keep flushed timeout packets. + * Pointer array used to keep flushed packets. * @param max_nb_out - * the element number of out. It's also the max number of timeout + * The element number of 'out'. It's also the max number of timeout * packets that can be flushed finally. * * @return @@ -208,10 +200,10 @@ uint16_t rte_gro_timeout_flush(void *ctx, * of a given GRO context. * * @param ctx - * pointer points to a GRO context object. + * GRO context object pointer. * * @return - * the number of packets in all reassembly tables. + * The number of packets in the tables. */ uint64_t rte_gro_get_pkt_count(void *ctx); -- 2.7.4