From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 5D6772C5E for ; Fri, 29 Dec 2017 04:53:30 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 28 Dec 2017 19:53:29 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.45,473,1508828400"; d="scan'208";a="17303891" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by fmsmga001.fm.intel.com with ESMTP; 28 Dec 2017 19:53:29 -0800 Received: from fmsmsx119.amr.corp.intel.com (10.18.124.207) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 28 Dec 2017 19:53:29 -0800 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by FMSMSX119.amr.corp.intel.com (10.18.124.207) with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 28 Dec 2017 19:53:27 -0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.159]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.213]) with mapi id 14.03.0319.002; Fri, 29 Dec 2017 11:53:26 +0800 From: "Chen, Junjie J" To: "Hu, Jiayu" , "dev@dpdk.org" CC: "Tan, Jianfeng" , "Ananyev, Konstantin" , "stephen@networkplumber.org" , "Yigit, Ferruh" , "Yao, Lei A" Thread-Topic: [PATCH v3 1/2] gro: code cleanup Thread-Index: AQHTevWIRlrBFaxkQUOR7JQZyoUd3KNZuxgw Date: Fri, 29 Dec 2017 03:53:25 +0000 Message-ID: References: <1513219779-100115-1-git-send-email-jiayu.hu@intel.com> <1513927544-97241-1-git-send-email-jiayu.hu@intel.com> <1513927544-97241-2-git-send-email-jiayu.hu@intel.com> In-Reply-To: <1513927544-97241-2-git-send-email-jiayu.hu@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiODJiZWQ0MTgtM2NkMi00ZjgxLTllMzctNjc2ZWVmYjc2NTZlIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjIuNS4xOCIsIlRydXN0ZWRMYWJlbEhhc2giOiJVbzAzMDhiR3ZocmgrMHhvOFB3RlIwNUZ2V2NGRlhTUlM3bEw4ZzA0aWtRVGNNWEJyTmx4dUpScFlGQTR6c3JzIn0= x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v3 1/2] gro: code cleanup X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Dec 2017 03:53:31 -0000 > -----Original Message----- > From: Hu, Jiayu > Sent: Friday, December 22, 2017 3:26 PM > To: dev@dpdk.org > Cc: Tan, Jianfeng ; Chen, Junjie J > ; Ananyev, Konstantin > ; stephen@networkplumber.org; Yigit, > Ferruh ; Yao, Lei A ; Hu, Ji= ayu > > Subject: [PATCH v3 1/2] gro: code cleanup >=20 > - Remove needless check and variants > - For better understanding, update the programmer guide and rename > internal functions and variants > - For supporting tunneled gro, move common internal functions from > gro_tcp4.c to gro_tcp4.h > - Comply RFC 6864 to process the IPv4 ID field >=20 > Signed-off-by: Jiayu Hu > --- > .../prog_guide/generic_receive_offload_lib.rst | 246 ++++++++------- > doc/guides/prog_guide/img/gro-key-algorithm.png | Bin 0 -> 28231 > bytes > lib/librte_gro/gro_tcp4.c | 330 > +++++++-------------- > lib/librte_gro/gro_tcp4.h | 253 > +++++++++++----- > lib/librte_gro/rte_gro.c | 98 +++--- > lib/librte_gro/rte_gro.h | 92 +++--- > 6 files changed, 518 insertions(+), 501 deletions(-) > create mode 100644 doc/guides/prog_guide/img/gro-key-algorithm.png >=20 > diff --git a/doc/guides/prog_guide/generic_receive_offload_lib.rst > b/doc/guides/prog_guide/generic_receive_offload_lib.rst > index 22e50ec..c2d7a41 100644 > --- a/doc/guides/prog_guide/generic_receive_offload_lib.rst > +++ b/doc/guides/prog_guide/generic_receive_offload_lib.rst > @@ -32,128 +32,162 @@ Generic Receive Offload Library > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D >=20 > Generic Receive Offload (GRO) is a widely used SW-based offloading > -technique to reduce per-packet processing overhead. It gains performance > -by reassembling small packets into large ones. To enable more flexibilit= y > -to applications, DPDK implements GRO as a standalone library. Applicatio= ns > -explicitly use the GRO library to merge small packets into large ones. > - > -The GRO library assumes all input packets have correct checksums. In > -addition, the GRO library doesn't re-calculate checksums for merged > -packets. If input packets are IP fragmented, the GRO library assumes > -they are complete packets (i.e. with L4 headers). > - > -Currently, the GRO library implements TCP/IPv4 packet reassembly. > - > -Reassembly Modes > ----------------- > - > -The GRO library provides two reassembly modes: lightweight and > -heavyweight mode. If applications want to merge packets in a simple way, > -they can use the lightweight mode API. If applications want more > -fine-grained controls, they can choose the heavyweight mode API. > - > -Lightweight Mode > -~~~~~~~~~~~~~~~~ > - > -The ``rte_gro_reassemble_burst()`` function is used for reassembly in > -lightweight mode. It tries to merge N input packets at a time, where > -N should be less than or equal to ``RTE_GRO_MAX_BURST_ITEM_NUM``. > - > -In each invocation, ``rte_gro_reassemble_burst()`` allocates temporary > -reassembly tables for the desired GRO types. Note that the reassembly > -table is a table structure used to reassemble packets and different GRO > -types (e.g. TCP/IPv4 GRO and TCP/IPv6 GRO) have different reassembly tab= le > -structures. The ``rte_gro_reassemble_burst()`` function uses the reassem= bly > -tables to merge the N input packets. > - > -For applications, performing GRO in lightweight mode is simple. They > -just need to invoke ``rte_gro_reassemble_burst()``. Applications can get > -GROed packets as soon as ``rte_gro_reassemble_burst()`` returns. > - > -Heavyweight Mode > -~~~~~~~~~~~~~~~~ > - > -The ``rte_gro_reassemble()`` function is used for reassembly in heavywei= ght > -mode. Compared with the lightweight mode, performing GRO in heavyweight > mode > -is relatively complicated. > - > -Before performing GRO, applications need to create a GRO context object > -by calling ``rte_gro_ctx_create()``. A GRO context object holds the > -reassembly tables of desired GRO types. Note that all update/lookup > -operations on the context object are not thread safe. So if different > -processes or threads want to access the same context object simultaneous= ly, > -some external syncing mechanisms must be used. > - > -Once the GRO context is created, applications can then use the > -``rte_gro_reassemble()`` function to merge packets. In each invocation, > -``rte_gro_reassemble()`` tries to merge input packets with the packets > -in the reassembly tables. If an input packet is an unsupported GRO type, > -or other errors happen (e.g. SYN bit is set), ``rte_gro_reassemble()`` > -returns the packet to applications. Otherwise, the input packet is eithe= r > -merged or inserted into a reassembly table. > - > -When applications want to get GRO processed packets, they need to use > -``rte_gro_timeout_flush()`` to flush them from the tables manually. > +technique to reduce per-packet processing overheads. By reassembling > +small packets into larger ones, GRO enables applications to process > +fewer large packets directly, thus reducing the number of packets to > +be processed. To benefit DPDK-based applications, like Open vSwitch, > +DPDK also provides own GRO implementation. In DPDK, GRO is implemented > +as a standalone library. Applications explicitly use the GRO library to > +reassemble packets. > + > +Overview > +-------- > + > +In the GRO library, there are many GRO types which are defined by packet > +types. One GRO type is in charge of process one kind of packets. For > +example, TCP/IPv4 GRO processes TCP/IPv4 packets. > + > +Each GRO type has a reassembly function, which defines own algorithm and > +table structure to reassemble packets. We assign input packets to the > +corresponding GRO functions by MBUF->packet_type. > + > +The GRO library doesn't check if input packets have correct checksums an= d > +doesn't re-calculate checksums for merged packets. The GRO library > +assumes the packets are complete (i.e., MF=3D=3D0 && frag_off=3D=3D0), w= hen IP > +fragmentation is possible (i.e., DF=3D=3D0). Additionally, it complies R= FC > +6864 to process the IPv4 ID field. >=20 > -TCP/IPv4 GRO > ------------- > +Currently, the GRO library provides GRO supports for TCP/IPv4 packets. > + > +Two Sets of API > +--------------- > + > +For different usage scenarios, the GRO library provides two sets of API. > +The one is called the lightweight mode API, which enables applications t= o > +merge a small number of packets rapidly; the other is called the > +heavyweight mode API, which provides fine-grained controls to > +applications and supports to merge a large number of packets. > + > +Lightweight Mode API > +~~~~~~~~~~~~~~~~~~~~ > + > +The lightweight mode only has one function ``rte_gro_reassemble_burst()`= `, > +which process N packets at a time. Using the lightweight mode API to > +merge packets is very simple. Calling ``rte_gro_reassemble_burst()`` is > +enough. The GROed packets are returned to applications as soon as it > +finishes. > + > +In ``rte_gro_reassemble_burst()``, table structures of different GRO > +types are allocated in the stack. This design simplifies applications' > +operations. However, limited by the stack size, the maximum number of > +packets that ``rte_gro_reassemble_burst()`` can process in an invocation > +should be less than or equal to ``RTE_GRO_MAX_BURST_ITEM_NUM``. > + > +Heavyweight Mode API > +~~~~~~~~~~~~~~~~~~~~ > + > +Compared with the lightweight mode, using the heavyweight mode API is > +relatively complex. Firstly, applications need to create a GRO context > +by ``rte_gro_ctx_create()``. ``rte_gro_ctx_create()`` allocates tables > +structures in the heap and stores their pointers in the GRO context. > +Secondly, applications use ``rte_gro_reassemble()`` to merge packets. > +If input packets have invalid parameters, ``rte_gro_reassemble()`` > +returns them to applications. For example, packets of unsupported GRO > +types or TCP SYN packets are returned. Otherwise, the input packets are > +either merged with the existed packets in the tables or inserted into th= e > +tables. Finally, applications use ``rte_gro_timeout_flush()`` to flush > +packets from the tables, when they want to get the GROed packets. > + > +Note that all update/lookup operations on the GRO context are not thread > +safe. So if different processes or threads want to access the same > +context object simultaneously, some external syncing mechanisms must be > +used. > + > +Reassembly Algorithm > +-------------------- > + > +The reassembly algorithm is used for reassembling packets. In the GRO > +library, different GRO types can use different algorithms. In this > +section, we will introduce an algorithm, which is used by TCP/IPv4 GRO. >=20 > -TCP/IPv4 GRO supports merging small TCP/IPv4 packets into large ones, > -using a table structure called the TCP/IPv4 reassembly table. > +Challenges > +~~~~~~~~~~ >=20 > -TCP/IPv4 Reassembly Table > -~~~~~~~~~~~~~~~~~~~~~~~~~ > +The reassembly algorithm determines the efficiency of GRO. There are two > +challenges in the algorithm design: >=20 > -A TCP/IPv4 reassembly table includes a "key" array and an "item" array. > -The key array keeps the criteria to merge packets and the item array > -keeps the packet information. > +- a high cost algorithm/implementation would cause packet dropping in a > + high speed network. >=20 > -Each key in the key array points to an item group, which consists of > -packets which have the same criteria values but can't be merged. A key > -in the key array includes two parts: > +- packet reordering makes it hard to merge packets. For example, Linux > + GRO fails to merge packets when encounters packet reordering. >=20 > -* ``criteria``: the criteria to merge packets. If two packets can be > - merged, they must have the same criteria values. > +The above two challenges require our algorithm is: >=20 > -* ``start_index``: the item array index of the first packet in the item > - group. > +- lightweight enough to scale fast networking speed >=20 > -Each element in the item array keeps the information of a packet. An ite= m > -in the item array mainly includes three parts: > +- capable of handling packet reordering >=20 > -* ``firstseg``: the mbuf address of the first segment of the packet. > +In DPDK GRO, we use a key-based algorithm to address the two challenges. >=20 > -* ``lastseg``: the mbuf address of the last segment of the packet. > +Key-based Reassembly Algorithm > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +:numref:`figure_gro-key-algorithm` illustrates the procedure of the > +key-based algorithm. Packets are classified into "flows" by some header > +fields (we call them as "key"). To process an input packet, the algorith= m > +searches for a matched "flow" (i.e., the same value of key) for the > +packet first, then checks all packets in the "flow" and tries to find a > +"neighbor" for it. If find a "neighbor", merge the two packets together. > +If can't find a "neighbor", store the packet into its "flow". If can't > +find a matched "flow", insert a new "flow" and store the packet into the > +"flow". > + > +.. note:: > + Packets in the same "flow" that can't merge are always caused > + by packet reordering. > + > +The key-based algorithm has two characters: > + > +- classifying packets into "flows" to accelerate packet aggregation is > + simple (address challenge 1). > + > +- storing out-of-order packets makes it possible to merge later (address > + challenge 2). > + > +.. _figure_gro-key-algorithm: > + > +.. figure:: img/gro-key-algorithm.* > + :align: center > + > + Key-based Reassembly Algorithm > + > +TCP/IPv4 GRO > +------------ >=20 > -* ``next_pkt_index``: the item array index of the next packet in the sam= e > - item group. TCP/IPv4 GRO uses ``next_pkt_index`` to chain the packets > - that have the same criteria value but can't be merged together. > +The table structure used by TCP/IPv4 GRO contains two arrays: flow array > +and item array. The flow array keeps flow information, and the item arra= y > +keeps packet information. >=20 > -Procedure to Reassemble a Packet > -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > +Header fields used to define a TCP/IPv4 flow include: >=20 > -To reassemble an incoming packet needs three steps: > +- source and destination: Ethernet and IP address, TCP port >=20 > -#. Check if the packet should be processed. Packets with one of the > - following properties aren't processed and are returned immediately: > +- TCP acknowledge number >=20 > - * FIN, SYN, RST, URG, PSH, ECE or CWR bit is set. > +TCP/IPv4 packets whose FIN, SYN, RST, URG, PSH, ECE or CWR bit is set > +won't be processed. >=20 > - * L4 payload length is 0. > +Header fields deciding if two packets are neighbors include: >=20 > -#. Traverse the key array to find a key which has the same criteria > - value with the incoming packet. If found, go to the next step. > - Otherwise, insert a new key and a new item for the packet. > +- TCP sequence number >=20 > -#. Locate the first packet in the item group via ``start_index``. Then > - traverse all packets in the item group via ``next_pkt_index``. If a > - packet is found which can be merged with the incoming one, merge them > - together. If one isn't found, insert the packet into this item group. > - Note that to merge two packets is to link them together via mbuf's > - ``next`` field. > +- IPv4 ID. The IPv4 ID fields of the packets, whose DF bit is 0, should > + be increased by 1. >=20 > -When packets are flushed from the reassembly table, TCP/IPv4 GRO updates > -packet header fields for the merged packets. Note that before reassembli= ng > -the packet, TCP/IPv4 GRO doesn't check if the checksums of packets are > -correct. Also, TCP/IPv4 GRO doesn't re-calculate checksums for merged > -packets. > +.. note:: > + We comply RFC 6864 to process the IPv4 ID field. Specifically, > + we check IPv4 ID fields for the packets whose DF bit is 0 and > + ignore IPv4 ID fields for the packets whose DF bit is 1. > + Additionally, packets which have different value of DF bit can't > + be merged. > diff --git a/doc/guides/prog_guide/img/gro-key-algorithm.png > b/doc/guides/prog_guide/img/gro-key-algorithm.png > new file mode 100644 > index > 0000000000000000000000000000000000000000..89cf427b3c7c406d7ddfb > 485d3cc5122b2206c1f > GIT binary patch > literal 28231 > zcmd?Qc|4n2`~RD^yTjJ%q@_x!c9)i-F;k)~rGufZA&SsSwWd%*43R{42Q_QADn > %)3 > zN=3DmjCF-8XxLy0M-ASDP%QA2`6_}yvu^L?J*bAIQ%&iVWN(N`L|?{%;1TG#bi*IM > g+ > zM_;ot7Z*D$1_FV^FJCgf0RnC2f %C > z5U4b9=3Di2S9!1v$pU%D9x0__hH{o6EhS3VR3LJVCtJ#QQ8GE?VQsLBc(IpS6qKP2 > 1t > zSMeX8B`5j~w2tXDgHMi@75Z84FTcVh#b-=3DiBwv bN_ z-yaon{D1tZ74P*}_ThfhYZrE^UTTO$@4c`oVsF{8huHb;7zY9&4|#-9jWpK$(3@7 > i > zoc4;b)tKk9Xt=3DnRes8vF^3{NMM5~y> nGS > z<6}iN3!|SuUXYN=3Dm;3o+cJxl0MA4m(I~tx8ng6`QW9)O!@n6mP|KBucu%yaZP{ > r5G > zR(mf_kLU5Xidwjqa)0Lz1WuMwXd~ogQuRf$b{D1htpEu+@bL@2 > t7{ > zjdfZvaRk+t%3i*jqE_sJo^M2ELh2S;R5%O1^+KMv`%ow|9v?ne7lP!nX~;}xzpE_- > zVOf3ajVdqdBGtIQnS1w?YikeQf>>l{W=3DCiV|6WSb(H}X@Z4A>a2z&G=3DU~V?^V7 > ^>@ > zjz@|6ix1Lm{V#}rB+16#f3j=3DV#@eWSz0RF&XH{G3k*x#0;pqO>y(Kwc<`G-*qV6A > q > z>b=3Di9NwhZ5FZa>KY!c0$Th>8o?oIAixuaZmow!$PzqyGY)3?ZQ=3D3eeBU)d(BwG* > pB > zf7w-@wOEL_IilNwxi(x{Td23YdQ%kh&AJD7WV%@6kBNOheGmJr`sVr_8}&OL > H7=3D#s > zU1X+fZ>T=3D<{xZUiiCGxXi}hhYeR;g + > z0ZR<`%H@}?uFa7Z<#ji-W(RuwY^+jjBQ1q{P1MVM9h9rS_F6vc`t4r>=3DWzhI0-l4 > w > z=3DH7Wzj)+otS*x6BS=3DTy^ayw(HY~n7iy+luVGzcwc+(sqIi|;U= x > z*daJK@FVQV+fqflax0M1sAgShdd_ zIy`|eF?-%t?qVa21H07E*DMb7)MRv{gm=3D@^)76Sjiv~-f=3Dfu$F!xwcvxJ5mOEXlW > ^ > z8qr^kUm#yqOM4Cn6 @vLL% > zd zH%JMy&0N2kWJ$=3D}+z ?ZbAr > z&z(ke5@U?s?vlAHba{pHyre(Ym3KAWXyjy`yN&vE Z0V > zo5S%mWbu_fLM9EjOWLzeY3|^Hv)_s2-1Hr<0#lCLmRKO6;V%dp+`o8%bVvM* > R{<|k > zS8~vN)oE_RJWO7c$k}S6k{u@xyOsRZhK}2S^K*aYsX`Fyb5HK|U!$?t!00x+^W; > C- > z|2p#w*W5q&Bjj`c&L3Ic+>d%FG;BS > z{oW%shA<|nDLK1q~F)^@c;Ex58wa4-4LWVJyeZeU6_;r#wlrk`?fY0_1Yh > w > z+202~^!;6wNC#Y7W$F9410r%z$22f(lG!zLBthCMs&j3qf6`8cGD47*XqcwP0R@ > kJ > zWs6j % > zs_R{_3A!!-MA#9VEzCl(i!2lT*d>6>zf^j-_rla2(}s(&*UKHL%0_85K5TVZb(s!J > zW1djE&{AGT_p8SshCY xy > zZYlhwJu^8!>!tlzx#c#}Ukfe$95aQhf`6NG!@2lgvYhI99d2 z1;sY2thzK3%fv;IrKE0QXl=3DR54$EV*kZW_!yQ4taA~|w3e`ONv&Enww|p0 > =3D > zmrvzT9*BJ3nwbikxbD#3yVpUWt7BhZY2pJ}?ZxoXEqe_APKYPe@dg!dB&esOgg > %$2 > zb;e8ei#%MZSec*N!0-pE$qp3KbP$&5GYhrcQ~k<-DxflqkJtbFD|;5 > z2h6YeMUP(bl`=3D!e#5d{|hZ*xc8WsI*JG6coFMm@wz>4tXmMa1O%etqw1$0`h > k+iW+ > z&vKxAj+nPre+SQHzLh*j`td~FCud4;u(#)`O;~{oX6*MfAXynDp=3Dqx{R8e0{(pla} > zNPQr;(R?E;%zq2$K`ZCbDuy+JRP}8sOvr;-)YS{~%M z3f0#>tG|=3Dkcu7Qa*iZ=3DXPnzq-d_imMArPqcxk}c`H-lj1NbHn}{GcgKSQcB9k}x06 > z4ih!-?QK?*V{RIX&`~e^>I8M=3Df5o_)Zvwq(%UYYH5=3De_j{G2FV+vU;sC14?aCHXF0 > zka^<{F(IqJce(Fs@q6Au>j#@|wa6|+ztc`D>Jy7OTx6%X%Lw#wwy8h7aeba5&oC > XB > zh*A0oIwQ4v-n92{V-H7o)Of$$$6|v`AkdfV-=3DG$>+(D=3D-1}T9^QOX2u`8ae)05ld; > z)F-c6bQ|?Yi zv12OKgGUcFO56gao`oAfAEx&R68~OGsk3{u33MxD=3D{H+2Z@M6B#B|s_3h3A0 > zmd_$ > zYKTRBu5&mr-?0=3D|3IhFoHf!xiBQmZINKBpUA=3D08@I{ha&t7#U&<}%**9KuN5qd > 9;; > z{jZHH*8(|*!bd+pvFv$WFB+1oZts-dhXbMJ?W=3D9Nv6Aa`5(N6*zF03PLdMYq6Ib > _% > z9CkU5YZ~llL!fN++Og5R2XqVVvcAA^oemXp`}}fd*F>Fn*bg@nbsoPj$LkNEjYQ`6 > z8$s)9tH=3Dm$vL6j7Tfat^W;VI8H-QplDFc|G4dJq@3X4s{ zy)e-yIAgo6+)8l4{4Ki&^gtinD{<2T7^de;y&C-5(*lup@IUMlg40pU#KSv*E5<5` > ziD$THf3WJg^Rz9-+%I_f)j#V4i#NuM=3D1?rQQpA)1S}*i7uI1y^jhtu0cMK&M^Wz1r > zODE<9^}G*wnGs$ucbSwq4(^YvUl~&;uwn*_s%$3)o A > z`>;X!+_MD~#QgbCr=3DyxL)IGD<)7L?nfx6Oe{J)r|^qy7^daDCs*6Vae_P > zk6X5M#^B zoa=3D&x440dA1=3Dpo9F|&2cBi@8LTNF@Sjy!X%_f9XnV>7qnoYyAqMx8%KJvN5Cj > =3Di|3 > z>8irLIdTFe7yhd9zD{zOs^0;+55X33)^g}2=3D5dkyIjtpB-QB30WmP?pr#ie;RMh?3 > zujLIg-JsBsRb99ji+X;pl > zUZ?Z7<;z*gq&u!k`6VSjH}BkR!_!(g4cUJDO8TRszRzhD6htaNekcm}XW+RcSdt57 > zs5gUi_nZp(sY87HEw;kvCrGSb`z#J&QCAUG4!NobGacw?9fEcdXQ%GoO6RE%3 > dwF> > zx7ed)rI!<3#`}y{B{&Gj;>55gM%HGYfoHfQa>Uzg-GJk`ZS9-37ll4Ot61%M?Z|xc > zkzpgj9^Yh&gvPcO!^@iOQk}y^a`lbXK@051j7hQy5|6>|R|v{cxwMkNpf8a28S? > |B > zM+(F=3D6kW7>b}wHP-j{bySoR;l*uu~Brn(Q!RjSIf4BPodsJfPC{sLuN`*Nxp<(5qF > zYX*&npunv{zX#JQ@?98lHqChM%MB1QnNiK{bs~DzFT? D7oC > z6h*QSU0Vw~8SI@-m{~>bTRwZY2Mg2GlJu`3Sk5v$$H2qpseC!KhU{m(rUQd-$ > ~u03 > z3k_QeH=3DOqgb!tgp}7Qn@@L++f`=3DN5(W#GWF5w > $*-I-` > zd!$2Hy70K0JBh}=3Dv5LJaDhIu6v~}!Ua^J&8FbSWomCcnbI( z*5zB*Vl^4wcN*ry`YLxk{lu~{)R{k=3DmI(bSPslqqtK>EaW>!f{y+h9wm24FJNePml > zAuj8Vo7swI7-%f8)8}j{e;k&Rktry0u{2~{>qnPw&a-~+GZd#(^6V59>G@ znF~HiSZI_vk~|&UfNj)xJohrWWW6YhWYTMmN#Nf3Fe%YJJ^P_`Z*jGzPP~`-kZT > zE-dujSVg~_(00`yZA%ak4E-5Vi)Dj(3q1X8CWqkIL~XgibC?4)UNSZe6xXa}Vrhct > z$R)n5%{d%fbp16atY)%o@;!BHHba}Sh~3wykR{KTrhX86tyyCj7UYDqA{*ExbiwS > O > zz?XD9vQ})Z3f0PCm}{B6mBFK-x64$?XTf)do-8btgL4q=3DN;pG +9 > zVI$^x?Wnae9m z{(byS;qNXJ`&y(PS>lgCnsMKXG|)GYTX7E1gF|-2t``NI-=3DdU~ADwGJ(3(f=3D5} zT!&_Zqj2>(b|GZS(G<Vc+h^U_GqammjUn5bqzM!hAI3Co?Lcdg0 > ~y) > zu3=3Dxx$4*@_%3p88%m+6nc%U`X7-6pa;J4o%{7_bhm+O?$T XH2T > zJmbBk!oIq#HJ6cL{wj$k7Y_>VD8F_o-c2DvDW!fJm}szWE=3D0w8HeAhwDM7zE3 > Qt`& > z;?3)8`3fL|esg|J@r8%=3DkcOX}{`7?gy~jISepEKvFMh-IWAQ_l4sz8>f4gS+d8KR* > zX?&FR>Ft}CH9lM=3DU%$}@GSDp97d3Hr0Rty&nHuC&?kHT6*!W-&sqj5LW!5PfL > IzL4 > z->n+;hJM|Y_@%_)MbYwEF3c5q>UHl~ZkkNym$ZAyod#O8Dmh~KItLvcyTyPt# > Phio > zG~y9%5sPVK#kK@sX%f_Xjc4Lx#Uqe%RmB&OD+X_Z8_=3D-QdpmRxUa>7veKOor > `Qbxj > z>zGQ6Ni(x>9ToTV8Ra~s#Z!Nqz zJDyHTV*LZQ#dc@b0TXD*ktoNR`(gT^#bQX0otJ > zDPU)x%v>eRtPd7y7t%0e^PoxU1XyIh!I%z}DEm$Wv-sYd6z9K--ygi@ICI3?aPk3P > zDVUaRQP;CPql?YLrt{85XiJ*Rhh;`b^8P4n=3DT^eXCQIUD7RWQ1i(xfmuPPcVCEKp` > zg%Qv?cUFgxPEO$Nvi1puQ1Cg#l*XOTm1t$_okaC986$3M>Cm`$=3DGZN(M0g$p > *?nMf > z%0b^qE;i$(;Dp1^HGzzl#>F%-q`9NBj~9c|A7tU4aJV>E{`SxnM> > p > zUbW9|OM#Wjo7q_F80F#|YFTA%=3D;lKR#uJB^TSj%HOeUzzURt}@>u*)E#5`@u9 > NgKl > zV7=3D6PFDiDL2u=3D(|R<=3D>9@H3xR9pN3?W+fq6@D6htQO!{({V&rMEbHnET=3D4U> > zU@7pQ>Ek41j|Tng1lZ&Co{^EhOs|Nn6vwunIj}1H!v|`TIWc=3D~lBB|0x~(Y{?-=3DXx > z4Ywp2W;KpQIa?a)=3DojM?JNsD%OI24E51G4FXIVs*q<;WMA)16=3DXO?&+rLj{hW > VDTJ > zmdPk@E?3A~kNi2+eU!-ap19>?sE;p;fpm{8DCg$S)a-2yIQp)|FMDnE?X1_mCyz` > | > zQxMZ)g?7VasYmwJ*Z68anh?3zN=3Dt#a6O{3Iv7G M > zRjcB>PDhR{9u4eTe2avHhBMqQdq)sryzJg{(7bh%ukZgZilGCWGAg{F(ZJBA_@~# > ` > zhuR?v8HMBqYn8|RnHj8Cm1_0P=3D!qJFl}#MC&6X^t(?aPA{1h54ms > O%K > z9WOXQVO^^f3OwrT>J@ZzyuQia-`lU)zf_8NR=3Dy9h^-2o#wW(Dro%kuSE^?b!)-C > W+ > zvBZgv8jtPKp}Mmg)q^)^s=3D$=3DH<76F25fKOG z0gtE1UfXE6F}`jO2fsjP>te`3K92JZbbRFz&l&X`{Tn29Mj*eUAy^#Y4gEgtHdQ4z > zeK+r&(m9(^XEawMUW$GWO=3DfzlEF1 WZ+ > zUw}34^3mf{zyGm#eZn&vgs)kb^+32q4xMb^FE+bZZ!WRUB;6W%GWQoPK_T > QL z1Ei+i7)F~gM(Xx*fhbI!JZc5JgTNdsKkGE4v-EiqZ6|-R?@&CDn}@uAO-MqCtIq{a > z@r^njzM&I#i#}6r-&1W_s6JlYVv)LCb;a4lCd?8K5zXb6e+p{MRI{C}5QrbCkD}kh > z8Ty~*)(}}3qQXmczJ=3DGjtB4 zbgu&0UZPb(m7(voVee?V>htU(q$ }FWj > z)JqC*cGby(ymEpo;HojZX!VtKZtuXH&&<6<@w&hDUA0L$esMftHi^@&ryNey{| > 0u9 > z84^{2Ky?#>pz38!h9&#Hm|awxJmP3AWkT 5jg@P > zXKt|LiByXQyZhOai3H~>MI#~cM=3D)#)lUGu;pNXXvz~zCY!hSz)Av_m!!TWRQ*p$_ > c > zBztm0SeY0O?IOKj)rNV{COgSg>2Yh< t* > zg^1Y{EbeQ62 zm@0g-2LpcQx0zhwa69bR pk@6 > za$kQqn6SlUEIw;njB?!*dS5v~puAK=3DRPk-Z)cBrzK^&BuM`|2%9SurmBx5hRy 7 > z&+i*4S&<`BN@E+mA9;kmh0FcRc1aJ7$=3D`?Fn%Kpg-)_0NlLp<&j-vM*ypOn1b1l > 7Q > zGtO#|w5~F&M8pzWt{ UsNz > zGw>km&@OkzKT~GkuO!pP1=3De|olUKE 3eM > z><7uq8#>hNpV!j;85uOgA&Cj)y2r#mJS&Yu*wKA-%KiF6?fOze(YPhd*t|}I!D49~ > z^SGNMsaKgme-~B=3Dnee{gZ6&w3A}h8Gm%q&pIj4kE^vfI)i!{jA9>58ewK|bqrjPP > 2 > z%eP*~433Ds@HK~m_8BM~co6^_bVU}a(y-DKb^>(3Bo8Ntn5*%&UpI{8zq3oVc > anVp > zM6GiIw(qiB>s*d<;5%?@CBxr|;jh!qg4~UadpK3`0ye45Wu(J > 3k > z2R$>x)7$dekINj%JF6%d(%`?Fxze|Kpe>P?5*=3DdHlGJvc7AclojK;8hc5(}7RzNJ3 > z3l1;OeVNg;lUuf%T29Rmm4x4vZ=3DuKJ4rVS+8qUI1gx4B^C*aZ)^M*16MTy$o1 > Wu%8 > z-*@Hh%a=3DoyUZ!p;c>;~SvkM;t>fcQn_@_Jlzlje6QRPus^VLShiryd6i86qeXS=3D!D > z5U>-pA8~ORe2t*BV}dkPwhq})^@VxZ;`%;DLsZK*a`IeKZDzGqX?&3sJMcU7N#g > Ah > z-N1Luz>`BMz@qQzdA$zaR!(T?yx4xw_ou)r{in8S`#g;FrRurr^u(0MP+W>%D9pu > D > zYan4hC;;X2QSfLOnF>F ?# > zwrY18=3Dsf+1j6t7fu0*9udLA)J*6opz<%ovb9T@7{dkw K > z3_xg!e~jM@x|I;Oj|JwQIV2rYY`T(4{MfN2E|ywc#F znrqKlglil19^pHPgCd!WK5T=3DK*&+6E`PC-&6DgN|pP~t1{@4UMV@{4*Ohthn9K > RQL > zD`J;HIPZJ4d> pl > z_?-akG3t zz09z?!%BYm|4WS`YfXieX^eM4*UWw&UTC-rzjxL > }9P > z9?<>qxP9Lf&!e_dFr5;S#%o`Uhu+p+Z?}ox3i>GPKSVeGw}pQdrM!w5i&oNc^74 > mm > zdXNVed29p% > Sqq > zvs|Bjgzr4Txd8vUSK`;tgSw(Zg;e5_rmqw`d`-R9c-4eazrGz-rE0UPUdu0ldl9hY > znOTY&Tuc~>PC53U17>o0&98Kf8o^<@lN7!-_O!-%*ZYh_2$PKG6_Zqg;0HLGI4 > PSw > z^l$U9pHm75G}&~>U(ROFiB!IBx5nI{{i_85cI~qaJC z?c)W(t3|F2thB z2|qaZd|O|rIaJ>)8pLCL&gP%lMNRrAAE(fLoUmCNGnNgOujICtT2l(_$pX!uYHZT > ~ > zuV!~AJbcJ%_H0k~c}vuOGi$zEgS0aFO-jAxSpGzX#l*uCi`nUQhQWZc_{+pSWi|h > k > zfB~|F&!*=3DF)hQWOqgZ>-gk+oO=3DYgYTlwmFvGi#2P6HS1>J$t@@mzVR)rVhFkF > O1U^ > zq)XPs;vtkipRp$=3D`VKBj7>q8h^iQMSy*pUo@YM2ttV3qUSq@718NC>_G=3D7vtXq > xF- > z_kwaMMSSPv&d -8e > z0`(qpx7hmCj}*640TKAp)cvc@U16LVNC8sKp5+<$iKefsG(=3DTyEB$LF;C%+KvSz)C > zDvY8&0u;wM;LcyUm+>7Jvo-Dw>P~iF{9^=3DkC-+dBQ|+4_ep1E8AZ^mQQ-^G)6; > nKO > z2QJLSBfd^Oyg73(0Hu8wzK_# # > zGn4txR2+@Yq=3DZZ%N7!p^v8syXZ&flf%O8Mp-HmPivRf{s&v>~?em?APV4u+3!s > YUu > ziPCws_RSL?5g&HY > p@~ > zpT_ayx{my9u{)!Bg|ei!(B|lJ z=3Djjq>tZW1R%w?}pS7GM9=3D@9R5*DU4gl0BtazIgcCK^#fns)_*H2{8*>=3D)RSlD|FI > @ > zn%s9a9EnM ^yt4 > zedFbQ`Rh^5=3DN%;sG9MU#12Wm;ElQVlJ{mOhdGo 6 > z$j;)+0*Aw;aYKp|@$M&$5|XCH9>e!i{wB169VWn?bMtO*{CmN%0+o-bh<%)P > cZ%?F > zOE5}DfrfsVM4`Zc1;pXb7r$d>HJ<77%mQSuuf><XGI1C_0l4ITe$TJ3WyI > zrYA|w)XjL zPjL(W<-Mj_ > Q > zBwFd19GMw{M6vQ0y&tKml743#5&OjUm&Sz#GC^=3D$>5Zy^ej+*O^w=3DSC?29J > mp+|P4 > zX|Hlk8G)YSe9oKI(V10;@g=3D>Nt7E$ho+kdv(o%7Q5cG(5CPk({=3D~x!$LTL%oCFE7 > w > zA@c?8oSpIK II& > z_l-lq2?=3D=3Drl13eN0eh*`4yFp1hZEiIe=3DT6xrO?@YoIiD<0kU&Of?2n~3PL0@_Yrc7 > zIVR38Zj2c2=3D6}~%K!V5kH49(GxloZbJpU_x_}YQLaSLwOVr@U28{ZjAgsRkCQ1ah > > > zSW%`qbuua3X{_<)rXISJPzu}>ZO@<_OSrw;Vz-0hwst*5*d|r^SUp9t%-!!kW8&cl > z=3Dbs#(?Ka4~zp#>e`r@n~uzAp#5v>2bJGD=3Drm+JHO7;OIdVWn&aa-D8iG^bOonD) > 19 > zL|Hv=3DeqE7#Y~p&xjbW=3Dkhe^Y#IEBy+Wz`BhWO+-7PpiKX`E$v c > zhqoM2ny?8Egivpm>nNg#nPki#}Zi{~F40U%nnh)kNLUuV!IKo{DR&??v > Z` > zKJELyh4pSgA(8mvPzN!a^IOw-QY(nXhdb&_KWCe^ufiRNNaDM-s^Th*m`88(9}b > fm > zz+WjRi0&!tQ6;sz(=3DOCgWX^6CWXQ!oeTY82(@IJ{<(JBzG>~_;daGh4_ZK1_16}K1 > z_H;=3DwR#u;rVLtJoDSVKU+F2bJxO2`aud$+H@TkwrKv`hr&v`w6;WF|F=3DW}4O2K > yjH > zZJ?u~0@`00Yp6zl(fyfOLIUUo-|i0oYoy+CGB>Mt6?OPOyN}#Fel|RJ#%|p?!Ybs=3D > zyQcF-uFgLV0N*E8+#72=3D9|Su}KALbFqE@9^vt%ALF{3$ODt;w#rdVy^K5C`%U& > FKd > zE#O%3p|Jf^8ThgL?vHE6%$<+Sv8_2G6YPX=3DW7XNu0(&4H`ObMVtXBUHW`A| > ByQ~tv > zOfu8k#Imf-4-!0>zq*fz93>Ua`PWpGe_gm{n3=3DZSWLI5(u&mY`j<>Uqp 4 > z=3DlZ4SZ9a@7n}*nPrgZ#SukAaNx}T^WDt)cEoq_g`_hhFNm$$}Amd2HQoCCSx&S > @z6 > z?+UyX@KIy_!hu9P=3Ds9oL-^cxZj*&ic^S=3D0hW4`~nmfF=3DwJ{5SFz%cD8kxg_T50IU > W > zk@si#xM|vr$VT8C4sjTfC8;!#876J77boWL572MKU*74;tQas~#WSnhR_wSHzK > 1Fr > znVylSmkQe}sn>A&-}%cklovljf8~g{2d5n>^^iny4r@G-6-sMPTO41CvLe;;4|E?1 > z#g(CLIy927KF-fu3peA*CuRDT8L2?<{CR5o*F!xE^IqlMi4K<9W>u&6`JuH4)m@`{ > zKT2=3D@EK@4Zvl7J1+Bdu|^0lrRwvZvjVijzIP}$@_(Zb}4O=3DBur{LmF&&fZ$T zw|ljk>?MN(dyaY!=3D?G3n(0UPM0* z > ziOnaz2>NW7ua=3DdH)k+h;?NZmGbk9Pfs>I_RZ8mlCqfK&NfLogH?jZlg_?Ry4P5P6 > T > z7G1ENvf1}-cDs;*PZ=3DDLv8mGaHLax(kcOp-^c|9ss2 > zg>v_ou Q > zs;)RSLP5Y*44x%sRm_BcHOzE86a&d^@iyE|Ll3^}ozKu@6GR&n$IX `j}j > z=3D_WE?O0-7LdAPlTd^&h7Xn(AO{caMZ)6zmh>@m15QThI)LF5^;Z`1$Mt;7Kxcn) > )4 > zB21CdF %w > zyNsr@IWQ8F^UZp$aw@Jej&QAQCGf)MJpX3+;GoMsKe0fDVF>T$n`mYUlT(3& > `9;d% > z70MD=3D8P_8kMYE2+f&amihV802s7Js#suIciu!ObT_N~krvHkwx`39p$eppy&RzG > Nv > zc_fwxjU2Hhq?x|2YGZ h > zo5pPdW2ZCXRYs4cua~Tp&UM$yMmUz9aScCEfQG-Rjx ;pFx > z4i5A|j>vQMjjLvlyD)KwWaj;$=3D>gvpD0v*m0&D~4CIqWyj$Gn4${lCn$-{`b!&l?$ > z*=3D>lH%8_1wq;NH~UHaF2@>@qV;lYtdwH?tkv9|+gXuj*6*_@Ape* > `9Q > zrF*+VtKLQNvw01-PV`77Ki_ie$zXo$tA{Z>nl+|3c>C|UR6A+-KFXQx`9m*z`-`Yo > zV&P*D@wy*{yXNTZ4~Y{MQkr_w=3D(S&CcIr9YHC9&M6uP!o15+BOcEH$eP4@6 > k_gdMj > zPdn~;I017>8v@aIiHU~6^GJmmp uyD-6 > zP-oB_xth9i54kp04x6yehtXdvF-*(I!!zFTNflMFD&gU$Bvd?ykic@DPX!c0bi}M_ > z_IK7_lR+@)h2Ox3eI>+iwe~j^4xB6RAXTAd6eZ%JW{Rq6-^x5xWyL?a0|~%^ &u > z`u&En)7_e#>8NrA^+NZc+G?`XQnG^9Na+Zy^?JB6d{}!{?&^$_>Teg}j2t_{^x4e7 > z2KWb{6a|)u?T0>LucXnD7q?x+Cv{G(Oy6~%YTRq1URK{SFzJ~op5*!JLaxD`z3}w > M > zY}NmM} > HLFEGO > zZ_bo5S|-UKTFnY{Z#Pn3%;>y_m`m3zspo{YizjwM1_jFglV4vUkCS-g$K8(u*nu&y > z0JixH*rwe2u@v0kyk!YkI5L2%W{0Yf$JYgZ5jj4hGyZlc0aBSzs>?txDze*MNtJfY > zP4|6nv2X8^UFLO;F_ECfQ&d0PGtUJi3j > k_ > zHhTmM3n}t2szPZZPlqeNvBF^_^9hiZ<(A0J0HA?pX9=3D4O$Rhypm0ox@VtShyX1 > $w& > zEtUP}W{qBnfjUd!2P&k-x2dL{Ipc03;wMNfBV7XgyM(o2X>|3Xl=3DXw#SAhPxl9pZ > x > zdILQeBKCpb=3DMs)Uc3bN1d$0*)Mkb6*?fjjz&L#Xe*>E)x!spkVV}#7Y+=3D0!6TjA^L > ztIH*Dj)>Zcu4vu+KcvI~8^Xaq>bAe#23u{|-nwlI=3D;M7z#Iy+~74=3D12_nixdOS3G4 > zMgF8OZhuvJpPP!B2FMdvUAC&9e_yOGGg#~en3f(YmiHn3 > WX z-wuIZeoVE#W3I{VdbEeM-dDp|O(5R>z6lh3F46^TTe<|77xXJut63Xv+V-GIlLt^I > zmUTf?qvdxpg$KhogMr>0_*Qp;;DGD9_VV=3Dqpedb}b>j!a=3Dhx>*?n2TvngW0o31 > 43^ > zet6I^+qF8Ih&Ea+30YH(GXi~mXkzr-D5@x>?DLp;!C2iKmc>qzCspzvwHgyq>mkP > F > z>vK2l04QfN&Mj-bGm4br;0mAgGun&#t)?ers-{6iy6k+4J{Eokpbv#_M34+7d*R| > v > z5NITd#61`BM|qiF)qt6(`=3Dh88Su3Nt$+$p8@u07MJrh1*vAs}~w > zZk;;GJGD(&QdyB59-Xq3n8*8xtcYa*@gM;fQOF&KL_mzi^Q6#Yu_EqMOJC%~g > ye42 > zukQe(D1wf28>eS$=3D3F)!^+hOQhciGW#wN)pidf1WZ5Q=3Dq%JhEGh+4X%J{@Dfn > 8l8v > z#~;qeO8nw+10~wGsapLa727QTKi^=3D^^2x;T21NXEwGNDb3RWHDDO?3$3&#k > ^TU+Wx > zYl?1`l|7p!f{a(o97Rm>KL~U>YQ9mbk522A@vHT(OB_U>HnXE#LqsD&1XkWYxj > hMR > zX_cQKCO9al-OTKV2)f%)TU1sBHecSfHaS5FF0kQ<4Pm-9 4@ > zKXFiUN2k8bD_XY}w!q!6d@vI_LE9}YI4vzyo+~{>Q?ntIb-x_+KMj8(x>_qTp?#pq > zN3|2+c^@aUQZ4J=3D>IkA^^Qgc4l0A>qYA5GwNpzo Q(8 > z0*C+>jNl9!tE4F1sjT}9DKlI?m4+0SAap*rCw1}+&90*mm*L}URn)&+Mz1?^@0{ > xj > z&ZG1_%CW`z)Btp=3Dh_=3DphNpY1nK!nNu?EYj zyt~uf;kY%dtE<4XrX_mXqz9Tgcm{Z4QEQLWpxokRoJrUR^p|X%u(%$3R$uVdFz > w)E > zw9Vc{CuM?iEBx5>GCMURvZvSUm!I-+m(x+t#R+c@dgL`GdlcJyJGoyuF+scXeqk > 4Q > z6f@C$&d+8<=3Ddc$W(=3D9 O> > z(z=3D4lAn}Gz(%FW`drncw5tqW~9#KUEd#Xy&9(Jr9_bT&b_w0JZ!O+WsxM7OP6- > ?AW > zO&~>0NH?d>CWjL)5#Q3dq=3D3BaiOVJ}bbAwvy0}V+AUXY|PfwkJ-xm4hZg#D`Fw > f_k > zli5aU9* e$ > z8lKyg8xfIl;y;tewrq!uF|UQ60cD@4!!koy?qkgGC2X-de2(!n?hE)bLQ;h5UI^?w > zW>NQ+HbFW#fnXHPsZb-sewoMMzD6}ORdNYX|1pN*C5_cVYvd%R;3!u$;o#3 > F^pqd6 > z@q09^QKj)o<8M1*GQXu;tZLU>$3jIXFmtZb{T%UJIf_>v3`l~-b?ltWG< C > ze=3DK`131^=3DmG_-!_?>X!4P!3bH!&8Ct(|Fj(HF0h3UxRNd3N3Y5Uid}t@DueU8>9 > GZ > z`d1`omtTxm{Cqi~?P9D5d47I9xzkQAo9a^RJJa`<%k@(K;4qCLomTowT~m50>N > 4Uw > zmS~Nz%c>z$$#ss6DP msIAH > zv!(t)ZbizaB?57Oak8T4OdbUQ)i|7pd=3D&xMTH682I_ChCb2Cr$!o@G0`^CCu3$@ > 2L > zB9A`aYf$?WqxLewFYSm2s-g=3D!jsP!H0ATfh{ROx`SW+Z@2_~ZSPJ5;1N~+Zv>=3DM > 1l > zP|Yc+>CI}KAu>oKy?=3DH6B-U2_YUjx`;WDEhwZ6*I)hC@|MHi|8mWUu)KxHj-xG > 8Ed > z$6*ww7>|b%d~euu816kxovV|>o+8W_MUPb{g%;O($=3Di#FV?B`wXTNGzblx^RX > ~-_6 > z$T<~UoZfbhjyo83Inu>_(hmEp?^ZoMDpSTw9VQ$ M4 > z=3D&-;*67u8Y7Eb@lpf>?=3DY+{}5cr(SP0r9Q;Fe91Kb%kWVo-5+TgJcaV|C_+Y{uhCZ > zakpV;3=3DTwjPq<{^?Sv z)c8IB19hW|?QIQt&o`zzZv#0Th{0?KiN>V~Pd3J~TpTE7l*_6Q(4#wD`%OEC@7O > SI > z=3D!{`ji(5WI=3D|GsguHAi?T|V~qocf4hHl&2KGG(l;?h69l4?A+aer~9yq|vetSBbCC > z=3DDrj4WeK_0W@BglkDZ*O|F1yYnX=3D-dtS@J|B0d}io^C%o=3DUg|3nRBXde;d3Bl% > d5> > zzb>399D6?Y*I4&uk06`5gM4CE7>9i|tBO$Tu?ci8-FUSv%N4^qJ5rI5w+FKFO|g1 > + > z9{|cN5E*6moH}YLMiB3^*b_2`7YBhTEvt(xXrl4@iekVKGl2Nr3e|STTO0xoZ-$?j > zF2@Kfq#go*u~yz((T3l~OntHXUpS}Q#Tf>PUnF0L;hs%gTVh+w3mR!xH@J*srW > okH > z$AO3?Vs4B%->GYE@O0OX#^Qo3EJ~u8O!EL}z5Z{;F*FrjJ0+m9(?cCJ2irv$3fOpp > zZw{h97*K?`@U=3DZ<8ms(0A5^=3Dshx1(`E}@5!eZ?Fa!v`~)vcU*g1A!8@3Wr~eC6B! > v > z>jO?1sw_=3DqtaD$XFo?V@r{_Ahfa1Og>IE;6KA4r!r@EcN0}JJ5qN;XBpA0B 9 > ztu ;a > zWz|6s;O4IP(35^xx zg*pqq07plbZ|T-R{fWtXzqxq*&>Js3QB|*GjP^eQp9465%N|(fnt3T|BLDDl(1W6 > E > zsQ-b!_Sa{LsG0IUolf77csI7X3tIRvw zQf1Z)9>LMW6DRXWR8I26DsZ1f*HSlT3wpu!j58IctKn0tF99#e@SSWMb%h^r7 > =3DWl=3D > zhTg<-*^k^GiRyR}aJ z{Em#mWdn~ePWW&&Y2#|uda13p8VF>=3Da>e|ZD`sY_n<0Vi!%ZN?2h8tXL9Evy > C4%CO > zuMY7$K_7cM8E=3D)_hy5u>(wm6sqUD*yNB>|?6hkmVoL$a>@M0$_K_H9Q8>? > Az4wNd) > zhOZcKBzI<$?7gS#Mk!ZkASPfOjIT)NM1>mg5<@|6mWYyVM(P#$mJ)ocr}L5Nu > 6)8q > ziL4ah(nS%3PpQ(p*Y0zS;VVLfEkKlkY|%Rz>|Q57n5fL_OU)@_@qzQnTl7P&zuW > A& > z|9EJbDb>sBNnHELxGAKi$f+%UeBZD0jR6#Kxv6Z}6W|c@P pL} > zQ|%(`VTsV@lvNg4UPu=3D9k)5Xkll`RRR*K|FIwf-(vrhIA>rV3Oi*1{@>N0oRa>0~9 > zDXf5jT@j~15x50JYsuo|vqglf8-0w27{pa(>$|o+g@48z#+6!dOGKUDbr0yGXF$G > - > zVt2k~?u>>h4I4OiO;IJ~?A-xMMyG|te=3DxQvJ;N<~YAV`V%B > z#T&Rsn27hN>0avYx>twG?F%fcCP0JBEayf!A1a^#{7h!9jfv{l`my6c`Ksb;8*2gE > zd2BYRV-3IXb*t#Orz+$tYgkr)#y> z%-n0mur6q6xV@9;p71Jn$wF?9&hXn^n5h58W8_Y5gkvGbK+=3DoqK)KZy7Ff8dv > #k{2 > zFVI2IR@lc%A1Cu)7Nvv{qx#-FjS>c6U%txKwT~@)?!Q > q > zDoA > =3DBf> > z$p&E9tgQ9RIKSO+*CS^V8v7I50QX;8M4a|{x4^3^o(x@k3ZSj(Wsa_wCxyV|M > @OTE > zzYDg#`O6n{=3DX>(K_I#JVS867HwU_osPX{IvXYVCXriS}~-DiIu&mmY46|OL?tSC`s > z!Qdlme87E5`_G7lr=3D$(>KOMIDHmRJg{ ~3r > zy^mHraE3K&bJ}U@Lrda}jWO}nQfcj__JX7 Zh > zLDGX2@zTo j{ > z%%z%4T6)*(ZNs_3yOHa^&)|c18*PWnmA7JIDVD6L!TrlNE;P};HGo6* S+ > zDL=3D9qHzs?K+`~DT_R=3D=3Deeq8m{YRTGParG^)KNL|3&0)K}OBxp6$9(ax*GpkV32x > @; > zPkMdD1?lZ7Fq$oim<^0@LjMdGzQ?%(dQT5QVWG%PJnTVbV)Z$1r+1o3-RQrj > 3WyWc > za3c+K5~%DNN-Mt9r;|p?Rzj2$qQZhY+1}nR$pKD*G9Ui*x*aIF93Q%SYB}3IrbX7 > E > zV?4k)rQ@OjHo zVH8}Rx;@@i+l=3Dmf#@T#mQol=3Dk#B1@gbaEj^X5w)ncWQ*7-Sp`phM1FbTdz > (U~x > zX?iZHdV7L)d#smDnhp7-hZ$THozlWnsjhmb|I1R8Ol1`GlH-yz$_DnPW|zei>%V* > + > zTWIapP_)aG)G;i6uX5+Hv#Y!s(V)}K=3D|glExD%q2)Y(s|P#zYDgD)uexqBs-;kqZ- > z_jKIhqr5!|X*an3Q)|s+NPZ*M@9e#?)CPE`d4?|iyHc4Sgk(bf1nh;IH@v5ubDvzz > zrIuI+o`z_5uPr9Upug{aLf5bzh`Xx#Z*&{S4|y(t#W#EAtbVJAR#-ebNwxKd&YR9_ > zjdZX>5Usi&6*CEGUs_mIvQ+O}{kpcKF$FyxXyH%>6>Me9lmM4Q!y?N*!4JYqcz > ynz > ztBy08za_?E{o<1chgc@Zs7wJ+$xCv}Q7ub{Dkhgo;4`%c?VOZx{+*=3D > 3 > z$-t9|1S}3wJg+BdC@Js&gw9swUU74Z<+I&oO0N|OEip *dy > zn+EwTprE)l?AU3q8mlxz>A-;tN`H5~P|x)E?9O{`y-R{Vc(77SryUyqh$!Bu z&2<%TC>UFaPk2f;6+uPtN#TJdw=3D^=3D1a@}}wEt=3DUqSn_dtCagV_)PP|Ej=3DdfEHQlQ > | > zbiUKNZ7x-gb#0`6&M<}*FjPXqs}a2|Y_XZfnGuKY5p{EQPSstJkxBe=3D%`P9c%f>h; > zkA#bms$$SmGZfQeebYNLrqi ue > zNMyZQELua?#h=3Dtc#^ZV1@`8(UjBLH#yQJ}?;23mUlU_;92yPenEmcirPKILSo7lI> > zRYpS)hfMC<^%_pA3~k4cl+4@JIF+s)x_@@L6PT$w{%n@O8)WAUbj*R<3Qx1H > zZ#is > zK5iR4MDZ9qoA{$)>5o=3D=3Dyj5H(sm-+2optWD-<^PARcBv_!Z1bQe5ZTpJ7JYJRuye > A > zw)M@EEj`a_Yu>z%e~7Z^M#oP3$BnJ|Y;Dqu7+YJ?8D+9cwdS{dE;K3~(NhfD$X-; > O > zgG&Ftk6Ny0D*bLiD{uOQdkktI?{tBSQF(V^NekBk=3D&Oa!E1gm6n5o > m > zXn~6!&s2+o`I6oocHBR6V(Xu33t<`5zDse^H0`pTlt+m9b-OeQmGzvgR&gD8OJ}h > G > z1DET|c92-^^S|cfZyXY>kg&z~DTHMZWJDZ=3DlI#ud8`I> R > z_LZJ~9$!Z>lIzx!55Q{nI@GLV5=3D06x2ey&v*)G4mybU?&y#mhWMPsyX+NcM;LF > c?e > zUkH1Mc%+0H>P;IzHn~X3yrVcL5pSj9P~PNTW0mVaeZ_AyU@>~KVz;)M!M4L8 > !dVR` > zv%jBMy6m<;IO_+baQ1u4gO6n|xuI&W4{b}4Ca$jNpNuZ~c;Hc(tc3i|B-Tamk#X > S* > z_1Sv?5Whs$w+hzv@|nGbYlac%c@HjhB~|u4f(*FqsRq}CBs5#K^DX%mXR=3D%7( > VA(C > zu$|gqzl2krdcVKsfOoT((ZjJUJIJ2lm$LP6>6YB!hN?`?^~~(P2T{v}E;kEr*mk&c > zi>Fs2^D5NmI%GMIQ)#QWdj~89XLtb_sy3DOsvBczbGF(ohCj+TwOTH}D4dNw+ > k#(B > zq9{^SCpa<`FlCU-sxtmfKCx1@J&a4#CP5%d`v@VQ09H0hhpV`qrgwdem%Z`;N> > |3A > z{r`%twcd4rZuQ+6c%6r|B$kp+w|Q*}YgjAR5L{bF43^a5h*#)R$p{?QmeM60kpx > ^6 > zw#!g42Tv$QS6R~f?We=3D)YZE;uV(B;o%Vfe}@D5w7oQ-^6rRs+n!{O~f2D)pi#ouP > 1 > zh_J4!fHG|S7O*ubhLw|2S1+AaI3*0p*DJ3yxiG}RJZ%sxsJJRZ_GSxKuL`R^I > z2uGooi$lvy+4T~OAhPcE;`FDI0B*{TVkRfniTit7nGUHx?>3Z#=3D^nx~RBAVL#Qx)e > z$I5CHeMVaA=3Dhfh% z6StHHtsF8O9qui8Bhz`YnexW&uKez!{!egEwp{uZEVMn_U!`Uyc3yM7J2c5Gcwe > XL > z(xBOSs{!1M2&j6lY&6&BxV3q;c35gx=3D}Z{#654Od#?&nK*z Oe > z%G~ClK>kUGMHFDX>k$+0@nC($oPC;Ug5>DfFgJ_qafm$kAhpKY5~ZxCG<{+l$L > P62 > zq#6^0*UB=3D+SO++!G*n6s06n|MfDtA^s2|>1A#aDPaLD%FTQfv@M > +@=3D~+ > z;WK>ZA@RJJRyo(OM1U1Q*Yt$%FGRS>TekOb07zycdg4(seSh0x#deKZdfI9IXZR > z7 > zBVN|!t|~u>KoYfIaz#C9ZDZPQEs|9G@(B-k!L > z4P;Y3hSpaSC|$O?h8kOc$qvra4mCnRZc1QXt)2QFAh)_r!xukzRuXWFyeLYTRUu > 6k > z(urTX#zix^&s0VnE30tqhD9Ryc$PP85tkftNg8|?VM0rRhz^w(3u5rkd&4H*6Y~x > y > zNt%qEYhbz_=3D_6sya42GxTJY%nfiwap(nW8h!WJ-`D80ufK9)lCa}L(utrR_|$m@J- > zfzjITf6Qy*+TBl{o!7=3D`yN-OV>?(Zp?(7xm zsu#dDbF;7oVC#?n6C}=3DAbmeMa$O|gv6SAU3_Mx?lG)Wm&Xe39$Y=3DP|* eWQ|8 > zVgg~ZjL(`O3jmgD&7;qqj7e6{+dq z;gky#tW_rSFwuR8XHwA?D*IA#x#n;Z{15UUNd#4JF~@m24O(1l(h>#smRZo(_z > eIy > z#El{{)Dw~cVxSA^dgqdCROlL>nMP+^v?H_*Oz4>!W|CcSv8fR 2 > zct-X$dIKcwMN!J{0WjG3mUMA*sev-oX7NUe{v=3D9+h9${!>cU;MG-B$c!Ewy=3DL7 > +4O > zkM*|(HTrjLy((C_APEhqY59KKMHab>7MUt(`9ZHI`Y@-EzmIjvqHjJ%u{JgoebFM > 8 > z;y0y#Yb 7 > zU&75)RFequ3C*kM;>o*)^2qjyC9uDd-C)NeWpyERm%jOOBXV5kgCvAkS`1OFP > H3!! > ztqi6Rz3C*%S7uQ~btVLEJn2A8VC}^PhoS{7fso|ud09GM>3}F|aC4|TD0YzCbNU > Va > z&?H9zYQNoh^7ZXX)^bU$+Bi)D`_2nF`peef*Uv=3D)NjCsTmTA|gbwcmTclJX=3DeU=3D2 > - > zRy_``v_w7Qfa<|4)+IB{kF5g@OX#&BhFJ1Jzd!!NXa{V*#!^1D- z%qW5mgr?lnyk?OwmOVd(n?#dV+ZT`LPo(Z+K51vM(Ongxa%M>>)IKlY$wN8V > al;GT > zvUXN>Wr8wdYLy8bi(8GY1Cn^40ojX?C>8N)5hTw;R#4i > z=3D zS+Y*qMuu!5*_knIl>KzdQbdR_7*lp; zkMHCACv)BReO=3DdmUDy4--tXu8b^YtWx!=3D3L+j_vCa;vjs-n266_z<6!`WYeSNBsm > c > zzh!F6q$FDmDZeA2vvms-#jFE9m!AEU>NIkz&MdyD VAn$1 > z86IAc>|6JCL&u69K&cTB > z* > z=3Dsy?o+{l_K&R2EJ7l3s>!tB-m?3=3DEzBV&&y8YXXl8m-B$$y^C}_I}coBz0M5{#0IM > z(s*3xdV#o7iK;q^8#OQzT9kHZM)+33jgbXhdG=3D(|;yA%lOO@p1j^wHshMj(}f6P< > P > zYVD+uaKC30<$_JhmKZb=3DuBVIL9 > V > zS*YKjp8GRj0_JrnbgVf%^~G*-Ow#n$BKA!G4b>oM?{Z$-m1b?nlYWELmgfFyH`n > hu > zVYA1mXAiNEd4()v?}}~>q^hm*AI4V5CldT`+F!-zB_%He3NwCXiDHXPM3Izh0=3DP > h4 > zu?gXu=3Dr$XB)gANI-2NtB!jZlVbFS-RMq8`;F-kSAgKpG9WW2iqmOJ(vW7c2eWFH > 0@ > zykB!S>*-esV95gvu(Pzr3z6;1)h; z>-d!_9+eOm(;}byeZSyOcGm1Y%>V$HLm@ y4~ > zaPZ`?geF9wU3GE2AGO*Dg-oCdp@u@V>k3+$14wx7Nh@I+j%RLV3;< NZ%^H > z!x#LH9OQYc0ety(xZYYAzA-(BYkv|GRu}(pPmYPL3Z3Ifv&;X`(!3;mUD#B(NzC4r > zVQrpzQT%Qj?o(~Mo9n~fOIHeqO0Dwy$T@al5_ieXSpYC{k}57}y<9hT!L+pMLRf > |Q > z?3*sqx%?8{m4rG!p V%!ROE > z;=3D@M&QB>V#55dHqwd(&Yc2~(o8}#Je(4aG=3DOH&~}wCl~g^6w=3DriY-C@)5w`rux; > D@ > zHoLL+-9;tCcnCl@xvli`I$>($fus{e(6g6%%>63Mo7vmV1Wa@?4xyStAr#{mp<`bp > z<6Id-S&NgMc%WnvW7?WNj;5r+#6+NCi9wq@{F$Fxc6}%3crZ-c2DOL)CeXWA > _~7wh > z1m7(m-*0kX8rBxtaAZj=3DdVkjX(kp6oZBi0;Z48%${(6iExeaHi9-Qj2L@itQ&P-wp > zQ0BN<0)Glga?m&+98kiYWP)eH+P9Nyp|=3DD}gO%LI(tN5MFQso&UaKX1lk5qPP > ?dqf > z0viA0p;+LiKZ)p;lR$rKR4-{oIMm!4H^r?ThN7)Hi|uFcg-++Cmd;CVjA85hBm;ny > z0zPa1SHBkj@c+0~c4{eh_GUawl#%5&g%e*4=3DH 82e > zS1}u54_bz4=3DnVaKzQwoX;Ju&t?|_dpp88N0q-*_Cx$M+J-~wA5w8ex(sJzW#`&A > |% > zfuk5BLf;6_@3}JYKyHPCSs9|LEFK9E0jPnH5 de > zlJUh(p(9ZpuY8qW{p7qm{=3Dk?LyQ{6&ggA*(8S{0Y)l*^D9{3hK!Wd%VZdYtJ4 > z > z>3s6m_ZMy7)=3DA!6*?)mKD{Ni&vf@wq8-clnR4&8r0JgJ^UPGS<>0OMN6>bm_ > XrsFO > z;1^a#lxs~erJWuTr}eD!oiKkE_I2D|EF5OAfb2`^@E?dQR9Kj{YtwUE_UExvksH*=3D > z<$+*f0Om}T#Os1b-ub^C=3D_(3h6{8vE*4P)a1DEJ$ > H > zYQi5qz~Fi{e)zlBjX*1lq5E%_=3DC!xJ#a~>%RlJ=3D)o2x?ph%zLoDa7wWKCq(-H#?L4 > znw^tJRFFn4>nO6%NAe}wFVrLe);WWLnJxJI>p7CmZy~oYmkB5r?@d$^;@G=3DM > n^_6} > zTCP-3WX8;0D!pAN=3DNr;;N5MDleCf=3D@ae)M4OEc|_f2aB`0>1kj0=3D28HanX5mnYi > =3D# > z+TMJ+xyfRMzJD!_sYxf-lM M > zxAw`g7krGlDXK5`Hb&^{ATm36`tFyw`wOdv&EIXr#$#y2B#vNJT=3Dk!l{P4!$&!G~h > zP3J;XWo{~K*jA2~eY&+rN6l`+kycx($VeY8bEzyS-Kn%e7*m_n^-lkCE+G875VU+ > O > zEnnz=3D|EsXe_SQ6fRO?6e?Wh;GbV35kR)R;4rt*hI#>@4RJ(|XH > z?UL@5`6tQE8>;Uxj$`wS%cUu1@?ClittA Eq1 > z*7fo M91~ > zK6t1&)PBuzY?_*+U$1%FZM3FrbA5aJ;Y*sWT!@%AS > F7_C? > zNPULZp-k$X&TPD?lG+5QKt091H*C5@5B7?gi0eDhE59BsSM*JNf4;7$N)G!n*} > Sor > zW>9~zi-A^|H;$7*s$U@Ny(01NW{Dk1!WhJpIkQ(@X}l5=3DLK^g1wPw%T$4RzIj > %Bsl > zniMbWYxdV9`vS5qkIXF95?Blyu13BT;bY^xwKEW0+v&fke||5z9)&;U-)+jCP}(mt > zti9vr>(IolcX0H@nEmr_nDeF{@NxhBi^e?9o%^dUfwU0jUrc&Po>LRz=3DBU9Xx2jC > % > zZw&rjom)oNFMH$2B ~sHx > zulvK{-Aoa=3D$q&@jhY1a(nl|UeBr^q}Rhk&@!-%^3{O7>-kWV+f8E>ZP%g@6{oHP > 8L > zlei*I#Ggi1|L*cDSD`LH)af$?q|@GUgi$^-Hl{3`_q`*UF1pQjiVsifh>qxucq7}i > zt4gtSuu=3D7n;^eJ=3D{)QVSz&S(~XNqvJWPT|PK|Q5mhBg5S5fb5=3D_X}4li`pN?gH#`C > zQFCcEzstiy!8!c5X7W#0S6#{u+q?}rqwatFc;$!kyK1NT{@r-9Xi_K>^|HUu?&1pW > zsEx1!Ty!vA(7?=3D#{h3NaW6n_6o%s&5Ak0J01LKQqX~n_N;vzyX`!`P~n3uRfUYsm > { > zqJX&h);N?h!R+ynJ%*P#>4zq3&uJa;O{MMcsn?;F98EV3i?jVLiFa^5y7?!XNBk(M > z%t8Ca=3DDtQiLs(4Ce#a+rL9)Wy%QwpJ6AMd3@BYrt{GR=3D-0`K*vpcEU+9mo%EVI > F4O > z<4X8l*S570UT;xB#>tMA&J->Litp3V4Q&5=3DaNaJsYu6%p|3-T*!7uwoPyzXYZ>o? > N > zqj^7S_^^riP18LS(uo(T#`{1Y#B42;W;@8OwhoI*t%TFL+M0bFka)-*;*)e#3O*gI > z#HOn)<-e2BK~4sWcLm8Lc{ 6 > z2F8W9t1zh{kNS6l0r*0kW?5kmGLyP*%XN9Vx&3~WZF;`q;8?Q6`O7=3DTGztHni > &qM> > zY&c!0m!J|Y8uk1&8G2=3DLg(2M0N(DO{Ba(7b_d}4A; zsF^*!B<#X?kT%P>;=3D=3DH@!bcsGU-uY!Ppkh~Qm}3zmP9^_Uwbz{R y@} > zG% 7GF;3p > z9FS&NdO)h`_kEu?UVd)wrMXV}l^5O|PKK9~(!Yl%v~|dRQ{#GF@}@}U-${H9cx > VvZ > z-hIaKZB|Y-DMR;x3j42duH*a><+AW0eVHodM%0_Q!X!hY9Y%dFZXsw0=3D0O?4k > 4(8I > zlkwx#^^b)_g(StbprHn0mjBe zwyjvs5qSSboU#Sw7bR!33JG`S=3DamMHavh`~($4oiz&x+09fHAvC&v}x0N|t|u{; > @u > zf)T09X6)w^@3Kva4_ioaQuvDaW}vzC-h(UiGHRJ+Z-b~^3?aT^)%H%viYH1+PGV > GI > z3!!AlS}822UonM$I|$}?-;Hr8K5{ul202yaf9LVYzf6eTvX!l|(O0e`<+{Sj^#hZa > zXHkr(JvA+->BR*|J0dGw^+YZS7cbi}b^QVSkLPNc$G9Gctg!Q=3D%|Ut)M`5{DR{$R > P > ziK7%9s>@<(+a4(fQb6E#(}n}J*!WIHZtyX&?8XMfR%xq^StAg+CUNIzAi(@yu-3}~ > zWFFo{4e0_AFgT=3DH7hTWQqC0Jsj-UrDo@>#aZxR4;Zn4!12q<>!aI=3D_Zl$-W9Pg)} > S > zP7Var$L4#JlK?0B^sKur5~#PSTZCxVjzo+0@vzek>vgx#Fj)RaY9{)tUB)oDJbb#_ > z2)Pj)&C*&aeGLW5b+8+25mW;E&Q6JbeVW^JCC%4 sA8o` > z#RLB1Zs1}E9xhE C@ > zd;hsat+{@E(vZrt!c+ zf~4$pcEA_!NS-F|hB-6V(jzulK^Z-QowKX350A}S2qBL7{a{nqwT9;_gHH4M > z$-0#>rER@XxV|R8zC=3Dt83WEb7xpm2CcJo;-Kt~jZAfvp+vpEyt4}(SLB7Ey1UR6>0 > zO6{5xPf(IKIiB_t%;Ydu_%HdY!PhaNbu&i+M-ofy!d|e-#eIaq-p zb&RaA!d=3DDk$XNSh{iaH|#VQP{{I > H > zyI^PB4{=3DH1>_#>=3DmI6;%Sn1;9S z+|#3Cy}g$>Ezy8#{XP)F57^)I?iGO z1~qS-$U@%|m|pH_G$rQY>#Z;MHE1^69Qlh=3Dr-5%+=3DcqgOgRz+ClI8NkCEF!u-D > >vW > z2BEi-GdQVRCDCveyGT2;3_Ekxb?PwrP^T_$+T2fiaQ&Rprtl?GPN$ayH&!vb > O7 > z6G0O|ix}VsL!%;x@n!9svX{@EEUE7V#EYN-PcVRVcqc4TyyPP{$%X8cUA3u^DR > Y8& > zuL!QNxZM)j^bOCdLGkQ9ZmgN_D~5RS@u e;{ > z#l_!;z+Pg81vA=3DZir`f4!n8eG*KORag#NG|9$%vU6>%8F=3D$0OTyhlo9-!xo_!RytU > zt48@OuzfDf-v!i;<;nGvdrwK6QebkkTps+2xOe1KRww|^06e(Abcej<58n1-wMh > Bz > zKLFhmBVJw70D^ {EP3 > z6fSAZNi?TKSPrwpKGg#%#TxA_jZWkATA#y#A+LP;l7t7rJiTdK z)y|I;1E#kvgY}6M4`F<(US+sPp2K~o6ANq7IRqfUCc_T##27;LM632wGAavonVeT > S > zjxAf7rBW(B{I^s?08*kV;9B0{x$YRd!vghZoZ80i758!cf > zT2CnsnN9+q+>@2c>(f}PlUYa>t3mEEw|6*YDNqj^@dE98ujMyxfw_fa3x`3$&F~ > xy > z#Zv2$H}Icqu*e9M@PMM^w&ZShcyi0}zGK#gLB+`v>viS>8!yL$r9qeP>NtPT(Nr > bQ > zLf%TT=3Dak+_J+J8y;fCst*QczX@#EuPw)O-Q+mMuDOGFruAk%9NXrM?YyX!Gjgs > qa3 > zkLMPx8OP;Q)X#~eR}V2`U(a`p74Xmj*S7iA5L@vWRbK+p*XqekpKfUzx2e`e^t4Y > 8 > zN%ma}lIs;=3Dle*w_+I}q9(o$ zL?(Brb>_8;qjFVIgK}fao{~M~ ^ > z@a%ZjPR!h~Yo{7CO!X9N#J5ISSEC80ZOYB+)V9Io;n#Oc;=3DOvi7F<;xJegb$etJ2N > zYk%io;|VCsixZk68#fpgj#;@er7uPARSzpLgN87u?AmfKzv+N1%@BBfEPfaz{OWj > x > z9$+k?zTA11Eq^bqZQfh8 z(*?X!ml+`yHRgnbS`Z_pL@ZjI+uJ&s{*YAp{0We%+YZeN55~E)Lcdd~J*mT9lq}WY > zFujFSIwF*{u!pzm7ON|ttkCj^91N{Cx);^Vxiti8^A2- zBy&p9#z}@YI64NO=3DQ{4+BPXjDAMBk6#H1J@5Pk`+$;UO8=3DDQtPQDsN&r03(0 > KmA&? > zUYtGd!!^Sobn}dIiz-|H6*6>X%5kyOf3a=3D~qkf_!gi+;*pP3VlBhU+zg0fP2_il ztL%JUTbSxv=3D62T}dx)_zjpN9%lP7H^q{eNS9hTuLMapH4ICc?o+Uu1~LOQ*W#m > TjV > zC;w4X84TTxvn)zUCYQ5VV%D~E?xo!Fm%+QB9?sUBI%z`uO7VGWBzoy7Dpfr& > k1m3% > zw~fesl9^CsTVT^*Qj<_NFv1jh zSW5OzFl zT?^4)t-{@`-ER3j8QLK{@A0gpulmDLIa98UTf8SD54~qCHi19lIK^#@#j`e~nCID` > zf U > z3}oo|c58g~3-uYGe*w{^%OYxQe;V}7{g429MoL8< Qc > zNYQug&2fV2HBg!R`9Lq(F24LAQaF6R{cPurThIX8WeU`o%TL JdM > zkXkPEyA@D(mpAi=3D#D1aD9p^!5jZNX*zwu+~yuWE=3DK|mw_*FQHK{$CX2Ph4jp > !>98o > z?r`pB3Vij%hHDT|Fl9H4H_QRbnAa!`duw@l)@qOSX=3D{^uNEjE=3D{LsGy)h$ANuy=3D; > & > z#yjs4bPWj9`rYC(s}`#yK-+>pHJb1?)`vGDf znqmoL0BnlWA~q(UwgJul3j}ke%Y@5H76c2WPxA|#6fE#ArhCMKwz7d4vaYzXp > opj% > z3Z`*J19??EByNM{KIed8elTi{6NSMXoOpORXCLlC4N?1&>_q~c4mq > UKm > zw*i$du+L2o7v`k<)_P(rIGHPh^_;fhI%}u5FxZ!l`fo@HJKo!sHD=3DHroq&mU&I|%{ > zShjA07lYlJfSB5tTgh}Ju8kYD#T|jc$}MK=3DxsqsB#o-92W-!ZpUjfr>IGg&bPP7Do > zDi7%qT{^LF@B!+5)onr~5zY%r67@Re)L}4gF2a8;k@whKw A@+{ > zaB%`K*o{s+dKU1~;XvZmwA=3Ducpceym+jYVdg>xKZHPZMFn6vg$i3SVJt-_;DV^!i > n > z!}6dKR>cY$oYHYd&Mz=3Dn-Qrov?~Ot6n5Awbbqhf-e8O`LmDn8h$=3DWYdSq0XDA > j_=3D( > zxgutQ1k`BZdC-`Yg_u)d)04hTH5{|uEFjKXx!T+h{Ixzg3ZTf6At=3D%bHZGvZUsjHP > zz#ti2c1kP(qP}Gq(09zQ_hkM$7(4dJo9mn;xT6zP2!03!og`rG`9$u%q8cS?7Yw$% > z6SAQSE^Y42ZbKGD@A1tI-dpI50>(iQ<#7Jl8Cnd30GA0c(9KC&yX7R_v`iegkWPq > 0 > z&sNxEPB+#2FQ;y9(_sHIx94n|pUGN_@g7?M>0Np4s5$ODA9aDTBHMXz^^xRv > t?_zJ > z?r=3D4h$b(o+`jZ{B9+64C?6G`KpcpQ&pgV0hG%63~um7f>tFS&A*WfTApkB``p)P!I > zwh2qwmA~%#w%2@H!c$0_U-lYf-~EI?yIV2IEC{k3{=3D?jQ_5W3d+*Iz=3D_(6`dB?Uv > z > zyHf}kKEv+$4J89+Qnqm#XWLP$Ii(nD{Ff$#U%V|VK=3D;_OrN{6ko?p1zWYq@qLE > D zjZr*l>03qmE}x{;UbQ^pq$A7W6%I|w<=3DptoWDsA`(3KFsY(u2z%DXrosM1UiPe> > hL > zpo^-!_ZU!|KvIz--q_~r@>HRT*osC<*s#g4#zbuqPZ9vJPOP)#kyWaGRPAML+(c?? > zTL57ujjZ%RBME3VSl5We`Al{?%wK^PyyP?NoiD=3Dk48J07+O5jIJHj8vCtP&ncFv< > % > zZrgWFa}~eC-OjOBNDr}x9I{_mg>fz?OUYUdtICv!iUV)9ksa5zTIaI~qRGE=3Dsp67b > z#7FCZmn@B43D>+6n|v|_a%zRA(&H#ftg~z{gz7%kjmoF+faWpit7!wWkYU0KN > f%YZ > zI64z?Ehk0X)2X&}h1Uc>;ZvfjC9}4MMSQ99(L$~kPwRDM-<-OUGvr$`TYH18^Nd > !{ > z@U<$&1ZB9SKSA*32|`}8LlU{c @A*OXD > zbRsN`eANe`nRD*CEOJ%VOjbRSuuz9D;SKen-+UR%uPIgIUMa%(n<*YnNci`7xNZ > 99 > zAJdO&<|jRFk1l&KTvJ@a!VKW){SHPdXoCC&Ls^lR=3Dr=3DDgLxz-VXNBiUt%5Ub dx > zi26bc`wX@FH`Gj3y1lSwf@+D%QS|&s95F{Z{XjAft=3DInbkL6vI>G{}2XOFtk4rA?> > z0V7)U@+Gp$vRvYu$?QaXn-uOdKZ&!(R+)^nDie&iR()cNA!L3*Z|!JlZJUt6*p0`+ > z3ALv6@HmgO96}!ur6UjoltbterWm)f3>iCobd&P^Dx=3D4Ry4Eu0w1v>G+VVM{< > NL1^ > zWyzyJtQK=3DC!p#9i|1}x6FlKt5uj<5byiLCff*ET_j3WPQg1t6C*;Ci zwhMJwQ1W{kLLCl5mAk5_K_?Xyfvte>Q>ml}Kr1&?SjH#LaCnEgC;bS2QhQ$2zG > kkG > zUZ!cTX{x8NYP$RwlOh`_sTYr%KFh{Tn1@+M!&#r`)mXkNcaPyfZCr3gd|(q z8y_t7@sWa_V(`%&|u?hMx`PM0sliu?S_{Va$ > jR > z%~|gdzr7lQP$HdgV7Kb>Sx?vt9o5!J8_FcTeq!YIeNlm#z^xJ*kjPcW1vhKU7u7~r > zzKNO<#N^XgE2Wa+biM012dgFnvn%<-WXpchmH~^x<>My2_e}`PQR7<<#}j2_ > Nv{ud > zm<&>Bm5tG-`jR`6`Ex!8=3Djo>CJQ<8uaEPSq_Ug+ewSIG#yqD&_C%TdxPqf5$qKbR > # > zW))CbG!maz9mDf!cy`Sc{JqU{y_V~99;cyQf%R@HxUzqRjtKrnyeYcs+_Tx%rI_rN > z?*6ih<)CK|ePW>ZNES`QBsfLQbkIUn*}i^jD7w zRbV_KbksOk^AvX{-1{+*jaqhG)bEiN_{x0r-RQn%Qq`pB&rfrzi%P=3D-&r0Q{A!w)w > z*&ZRk3;L(^;A8wz^CaUWTC)1GQ3KA5LWmoQzkW97bvcn?t`EI6Z*&DiZt*$!RIZ > x$ > zu+#*Fd0+tFdQ3$kk}2EA3igZ;YcR>!Nnn*C-2$14y}IJB`tjNy9b4!@ zoPkp1=3D=3DPXwa` znFyTm{5VcAFSn+TWxS&Fo?H1ae}LQk>L)P!HHls!DznOe%?lM3hl=3D5pM?0?4hk > 1W~ > zu&s@3!QlMol$T2{jlD!GIkK18Hnb-xro@4g*BFzf9v@pBCDy@i4aoKepNmL+R} > bwY > z%Br9m>J}v~hKolMvg3R9qp~{8Pve!ZTwqr2#+4D2!?5Y)3bnRn_SV;}fTPbqckf}0 > zT=3D%1X3Gp5W+n47i^6`$!-Hv$19?iWr4R&L*%YnjN1v@gC*`BZ4333dShArnqf6 > VA( > z2ODZ%D?e5?G!yh!rRbw z*-#hnm8HAajR8Ii3DXM94rcdcXnf@Fje* > z$uzIgkwtpy%E%(z#=3D}!%$wFH*`GJ$`AUfV&ChWtKu+MF!Z<-IcmTE M > zh2Y5N_x+^8&ZdQH>_+c}3=3D*&!@saTt_MFG-eUoiUheG9F7We?VB > MQ<} > z-hKEIaNx)PRh9Es_ZdYwedv}`FYc<^yeQz{){jt>`hSf90vyTEF&^yZ3H-+Qp7CbL > zvf8#~`uy-Qllsvh)o1gUe><7oKW2Q^cLaSil>j`j$m*NdFpaAqLC!Ml{LlaRjW=3Dxt > z;B%GabxS%TZ$?f&%2H7L9ZD}UcU+wUdI4dR&(jNl@wtm=3D0gA~XN{baWR@U > FGq;A=3Dk > zSWKt*j1c$RyM2ED7>`I0&Sr0)qz!Y_Z8v6;*7d?Hz$2O6g8LyCFe#A$V)koEroEeH > zZvilB1HrZeuXt`BZxF}@nLs|eX7Z!;T*5w)>! > zIm{p2b_5N!2DVTVe+g;^yWY7GCg20D&g$MOR*DcEftg>u!&;@y;GJ;_-~_R*B& > WJn > z?&NS?Vqy3}Xx9GN!(58s@tr<^Y8%)h%zY*(^`(S0;l$T<&WAh++21Z`0(}4a} > zXkzvi zOH2B}fg5ZblDoR`z$OBAdvEzUJUo_H+M%^%mrd>h*04y?J+KMJkA4@A$mZK > RYXmx; > zV`d}qieoPfmePb=3Dr{}TZDW6ulxZZM6^z6xGQBWV+lyg2k9B35UvLc4LrH;ig*b76 > X > z{fDYSFsBC$cB->eoIZqs{Q(uX0`LrZMGtUh$^~c#ws9&2dm1F2dq_S{+jRDUz7RLu > s@&`)&Hy$?zZu>uSyNlixoQ=20 > literal 0 > HcmV?d00001 >=20 > diff --git a/lib/librte_gro/gro_tcp4.c b/lib/librte_gro/gro_tcp4.c > index 61a0423..53747d4 100644 > --- a/lib/librte_gro/gro_tcp4.c > +++ b/lib/librte_gro/gro_tcp4.c > @@ -34,8 +34,6 @@ > #include > #include > #include > -#include > -#include >=20 > #include "gro_tcp4.h" >=20 > @@ -72,20 +70,20 @@ gro_tcp4_tbl_create(uint16_t socket_id, > } > tbl->max_item_num =3D entries_num; >=20 > - size =3D sizeof(struct gro_tcp4_key) * entries_num; > - tbl->keys =3D rte_zmalloc_socket(__func__, > + size =3D sizeof(struct gro_tcp4_flow) * entries_num; > + tbl->flows =3D rte_zmalloc_socket(__func__, > size, > RTE_CACHE_LINE_SIZE, > socket_id); > - if (tbl->keys =3D=3D NULL) { > + if (tbl->flows =3D=3D NULL) { > rte_free(tbl->items); > rte_free(tbl); > return NULL; > } > - /* INVALID_ARRAY_INDEX indicates empty key */ > + /* INVALID_ARRAY_INDEX indicates an empty flow */ > for (i =3D 0; i < entries_num; i++) > - tbl->keys[i].start_index =3D INVALID_ARRAY_INDEX; > - tbl->max_key_num =3D entries_num; > + tbl->flows[i].start_index =3D INVALID_ARRAY_INDEX; > + tbl->max_flow_num =3D entries_num; >=20 > return tbl; > } > @@ -97,116 +95,15 @@ gro_tcp4_tbl_destroy(void *tbl) >=20 > if (tcp_tbl) { > rte_free(tcp_tbl->items); > - rte_free(tcp_tbl->keys); > + rte_free(tcp_tbl->flows); > } > rte_free(tcp_tbl); > } >=20 > -/* > - * merge two TCP/IPv4 packets without updating checksums. > - * If cmp is larger than 0, append the new packet to the > - * original packet. Otherwise, pre-pend the new packet to > - * the original packet. > - */ > -static inline int > -merge_two_tcp4_packets(struct gro_tcp4_item *item_src, > - struct rte_mbuf *pkt, > - uint16_t ip_id, > - uint32_t sent_seq, > - int cmp) > -{ > - struct rte_mbuf *pkt_head, *pkt_tail, *lastseg; > - uint16_t tcp_datalen; > - > - if (cmp > 0) { > - pkt_head =3D item_src->firstseg; > - pkt_tail =3D pkt; > - } else { > - pkt_head =3D pkt; > - pkt_tail =3D item_src->firstseg; > - } > - > - /* check if the packet length will be beyond the max value */ > - tcp_datalen =3D pkt_tail->pkt_len - pkt_tail->l2_len - > - pkt_tail->l3_len - pkt_tail->l4_len; > - if (pkt_head->pkt_len - pkt_head->l2_len + tcp_datalen > > - TCP4_MAX_L3_LENGTH) > - return 0; > - > - /* remove packet header for the tail packet */ > - rte_pktmbuf_adj(pkt_tail, > - pkt_tail->l2_len + > - pkt_tail->l3_len + > - pkt_tail->l4_len); > - > - /* chain two packets together */ > - if (cmp > 0) { > - item_src->lastseg->next =3D pkt; > - item_src->lastseg =3D rte_pktmbuf_lastseg(pkt); > - /* update IP ID to the larger value */ > - item_src->ip_id =3D ip_id; > - } else { > - lastseg =3D rte_pktmbuf_lastseg(pkt); > - lastseg->next =3D item_src->firstseg; > - item_src->firstseg =3D pkt; > - /* update sent_seq to the smaller value */ > - item_src->sent_seq =3D sent_seq; > - } > - item_src->nb_merged++; > - > - /* update mbuf metadata for the merged packet */ > - pkt_head->nb_segs +=3D pkt_tail->nb_segs; > - pkt_head->pkt_len +=3D pkt_tail->pkt_len; > - > - return 1; > -} > - > -static inline int > -check_seq_option(struct gro_tcp4_item *item, > - struct tcp_hdr *tcp_hdr, > - uint16_t tcp_hl, > - uint16_t tcp_dl, > - uint16_t ip_id, > - uint32_t sent_seq) > -{ > - struct rte_mbuf *pkt0 =3D item->firstseg; > - struct ipv4_hdr *ipv4_hdr0; > - struct tcp_hdr *tcp_hdr0; > - uint16_t tcp_hl0, tcp_dl0; > - uint16_t len; > - > - ipv4_hdr0 =3D (struct ipv4_hdr *)(rte_pktmbuf_mtod(pkt0, char *) + > - pkt0->l2_len); > - tcp_hdr0 =3D (struct tcp_hdr *)((char *)ipv4_hdr0 + pkt0->l3_len); > - tcp_hl0 =3D pkt0->l4_len; > - > - /* check if TCP option fields equal. If not, return 0. */ > - len =3D RTE_MAX(tcp_hl, tcp_hl0) - sizeof(struct tcp_hdr); > - if ((tcp_hl !=3D tcp_hl0) || > - ((len > 0) && (memcmp(tcp_hdr + 1, > - tcp_hdr0 + 1, > - len) !=3D 0))) > - return 0; > - > - /* check if the two packets are neighbors */ > - tcp_dl0 =3D pkt0->pkt_len - pkt0->l2_len - pkt0->l3_len - tcp_hl0; > - if ((sent_seq =3D=3D (item->sent_seq + tcp_dl0)) && > - (ip_id =3D=3D (item->ip_id + 1))) > - /* append the new packet */ > - return 1; > - else if (((sent_seq + tcp_dl) =3D=3D item->sent_seq) && > - ((ip_id + item->nb_merged) =3D=3D item->ip_id)) > - /* pre-pend the new packet */ > - return -1; > - else > - return 0; > -} > - > static inline uint32_t > find_an_empty_item(struct gro_tcp4_tbl *tbl) > { > - uint32_t i; > - uint32_t max_item_num =3D tbl->max_item_num; > + uint32_t max_item_num =3D tbl->max_item_num, i; >=20 > for (i =3D 0; i < max_item_num; i++) > if (tbl->items[i].firstseg =3D=3D NULL) > @@ -215,13 +112,12 @@ find_an_empty_item(struct gro_tcp4_tbl *tbl) > } >=20 > static inline uint32_t > -find_an_empty_key(struct gro_tcp4_tbl *tbl) > +find_an_empty_flow(struct gro_tcp4_tbl *tbl) > { > - uint32_t i; > - uint32_t max_key_num =3D tbl->max_key_num; > + uint32_t max_flow_num =3D tbl->max_flow_num, i; >=20 > - for (i =3D 0; i < max_key_num; i++) > - if (tbl->keys[i].start_index =3D=3D INVALID_ARRAY_INDEX) > + for (i =3D 0; i < max_flow_num; i++) > + if (tbl->flows[i].start_index =3D=3D INVALID_ARRAY_INDEX) > return i; > return INVALID_ARRAY_INDEX; > } > @@ -229,10 +125,11 @@ find_an_empty_key(struct gro_tcp4_tbl *tbl) > static inline uint32_t > insert_new_item(struct gro_tcp4_tbl *tbl, > struct rte_mbuf *pkt, > - uint16_t ip_id, > - uint32_t sent_seq, > + uint64_t start_time, > uint32_t prev_idx, > - uint64_t start_time) > + uint32_t sent_seq, > + uint16_t ip_id, > + uint8_t is_atomic) > { > uint32_t item_idx; >=20 > @@ -247,9 +144,10 @@ insert_new_item(struct gro_tcp4_tbl *tbl, > tbl->items[item_idx].sent_seq =3D sent_seq; > tbl->items[item_idx].ip_id =3D ip_id; > tbl->items[item_idx].nb_merged =3D 1; > + tbl->items[item_idx].is_atomic =3D is_atomic; > tbl->item_num++; >=20 > - /* if the previous packet exists, chain the new one with it */ > + /* If the previous packet exists, chain them together. */ > if (prev_idx !=3D INVALID_ARRAY_INDEX) { > tbl->items[item_idx].next_pkt_idx =3D > tbl->items[prev_idx].next_pkt_idx; > @@ -260,12 +158,13 @@ insert_new_item(struct gro_tcp4_tbl *tbl, > } >=20 > static inline uint32_t > -delete_item(struct gro_tcp4_tbl *tbl, uint32_t item_idx, > +delete_item(struct gro_tcp4_tbl *tbl, > + uint32_t item_idx, > uint32_t prev_item_idx) > { > uint32_t next_idx =3D tbl->items[item_idx].next_pkt_idx; >=20 > - /* set NULL to firstseg to indicate it's an empty item */ > + /* NULL indicates an empty item. */ > tbl->items[item_idx].firstseg =3D NULL; > tbl->item_num--; > if (prev_item_idx !=3D INVALID_ARRAY_INDEX) > @@ -275,53 +174,33 @@ delete_item(struct gro_tcp4_tbl *tbl, uint32_t > item_idx, > } >=20 > static inline uint32_t > -insert_new_key(struct gro_tcp4_tbl *tbl, > - struct tcp4_key *key_src, > +insert_new_flow(struct gro_tcp4_tbl *tbl, > + struct tcp4_flow_key *src, > uint32_t item_idx) > { > - struct tcp4_key *key_dst; > - uint32_t key_idx; > + struct tcp4_flow_key *dst; > + uint32_t flow_idx; >=20 > - key_idx =3D find_an_empty_key(tbl); > - if (key_idx =3D=3D INVALID_ARRAY_INDEX) > + flow_idx =3D find_an_empty_flow(tbl); > + if (flow_idx =3D=3D INVALID_ARRAY_INDEX) > return INVALID_ARRAY_INDEX; >=20 > - key_dst =3D &(tbl->keys[key_idx].key); > + dst =3D &(tbl->flows[flow_idx].key); >=20 > - ether_addr_copy(&(key_src->eth_saddr), &(key_dst->eth_saddr)); > - ether_addr_copy(&(key_src->eth_daddr), &(key_dst->eth_daddr)); > - key_dst->ip_src_addr =3D key_src->ip_src_addr; > - key_dst->ip_dst_addr =3D key_src->ip_dst_addr; > - key_dst->recv_ack =3D key_src->recv_ack; > - key_dst->src_port =3D key_src->src_port; > - key_dst->dst_port =3D key_src->dst_port; > + ether_addr_copy(&(src->eth_saddr), &(dst->eth_saddr)); > + ether_addr_copy(&(src->eth_daddr), &(dst->eth_daddr)); > + dst->ip_src_addr =3D src->ip_src_addr; > + dst->ip_dst_addr =3D src->ip_dst_addr; > + dst->recv_ack =3D src->recv_ack; > + dst->src_port =3D src->src_port; > + dst->dst_port =3D src->dst_port; >=20 > - /* non-INVALID_ARRAY_INDEX value indicates this key is valid */ > - tbl->keys[key_idx].start_index =3D item_idx; > - tbl->key_num++; > + tbl->flows[flow_idx].start_index =3D item_idx; > + tbl->flow_num++; >=20 > - return key_idx; > + return flow_idx; > } >=20 > -static inline int > -is_same_key(struct tcp4_key k1, struct tcp4_key k2) > -{ > - if (is_same_ether_addr(&k1.eth_saddr, &k2.eth_saddr) =3D=3D 0) > - return 0; > - > - if (is_same_ether_addr(&k1.eth_daddr, &k2.eth_daddr) =3D=3D 0) > - return 0; > - > - return ((k1.ip_src_addr =3D=3D k2.ip_src_addr) && > - (k1.ip_dst_addr =3D=3D k2.ip_dst_addr) && > - (k1.recv_ack =3D=3D k2.recv_ack) && > - (k1.src_port =3D=3D k2.src_port) && > - (k1.dst_port =3D=3D k2.dst_port)); > -} > - > -/* > - * update packet length for the flushed packet. > - */ > static inline void > update_header(struct gro_tcp4_item *item) > { > @@ -343,84 +222,99 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, > struct ipv4_hdr *ipv4_hdr; > struct tcp_hdr *tcp_hdr; > uint32_t sent_seq; > - uint16_t tcp_dl, ip_id; > + uint16_t tcp_dl, ip_id, frag_off, hdr_len; > + uint8_t is_atomic; >=20 > - struct tcp4_key key; > + struct tcp4_flow_key key; > uint32_t cur_idx, prev_idx, item_idx; > - uint32_t i, max_key_num; > + uint32_t i, max_flow_num; > int cmp; >=20 > eth_hdr =3D rte_pktmbuf_mtod(pkt, struct ether_hdr *); > ipv4_hdr =3D (struct ipv4_hdr *)((char *)eth_hdr + pkt->l2_len); > tcp_hdr =3D (struct tcp_hdr *)((char *)ipv4_hdr + pkt->l3_len); > + hdr_len =3D pkt->l2_len + pkt->l3_len + pkt->l4_len; >=20 > /* > - * if FIN, SYN, RST, PSH, URG, ECE or > - * CWR is set, return immediately. > + * Don't process the packet which has FIN, SYN, RST, PSH, URG, ECE > + * or CWR set. > */ > if (tcp_hdr->tcp_flags !=3D TCP_ACK_FLAG) > return -1; > - /* if payload length is 0, return immediately */ > - tcp_dl =3D rte_be_to_cpu_16(ipv4_hdr->total_length) - pkt->l3_len - > - pkt->l4_len; > - if (tcp_dl =3D=3D 0) > + /* > + * Don't process the packet whose payload length is less than or > + * equal to 0. > + */ > + tcp_dl =3D pkt->pkt_len - hdr_len; > + if (tcp_dl <=3D 0) > return -1; >=20 > - ip_id =3D rte_be_to_cpu_16(ipv4_hdr->packet_id); > + /* > + * Save IPv4 ID for the packet whose DF bit is 0. For the packet > + * whose DF bit is 1, IPv4 ID is ignored. > + */ > + frag_off =3D rte_be_to_cpu_16(ipv4_hdr->fragment_offset); > + is_atomic =3D (frag_off & IPV4_HDR_DF_FLAG) =3D=3D IPV4_HDR_DF_FLAG; > + ip_id =3D is_atomic ? 0 : rte_be_to_cpu_16(ipv4_hdr->packet_id); > sent_seq =3D rte_be_to_cpu_32(tcp_hdr->sent_seq); >=20 > ether_addr_copy(&(eth_hdr->s_addr), &(key.eth_saddr)); > ether_addr_copy(&(eth_hdr->d_addr), &(key.eth_daddr)); > key.ip_src_addr =3D ipv4_hdr->src_addr; > key.ip_dst_addr =3D ipv4_hdr->dst_addr; > + key.recv_ack =3D tcp_hdr->recv_ack; > key.src_port =3D tcp_hdr->src_port; > key.dst_port =3D tcp_hdr->dst_port; > - key.recv_ack =3D tcp_hdr->recv_ack; >=20 > - /* search for a key */ > - max_key_num =3D tbl->max_key_num; > - for (i =3D 0; i < max_key_num; i++) { > - if ((tbl->keys[i].start_index !=3D INVALID_ARRAY_INDEX) && > - is_same_key(tbl->keys[i].key, key)) > + /* Search for a matched flow. */ > + max_flow_num =3D tbl->max_flow_num; > + for (i =3D 0; i < max_flow_num; i++) { > + if ((tbl->flows[i].start_index !=3D INVALID_ARRAY_INDEX) && > + is_same_tcp4_flow(tbl->flows[i].key, key)) > break; > } >=20 > - /* can't find a key, so insert a new key and a new item. */ > - if (i =3D=3D tbl->max_key_num) { > - item_idx =3D insert_new_item(tbl, pkt, ip_id, sent_seq, > - INVALID_ARRAY_INDEX, start_time); > + /* > + * Fail to find a matched flow. Insert a new flow and store the > + * packet into the flow. > + */ > + if (i =3D=3D tbl->max_flow_num) { > + item_idx =3D insert_new_item(tbl, pkt, start_time, > + INVALID_ARRAY_INDEX, sent_seq, ip_id, > + is_atomic); > if (item_idx =3D=3D INVALID_ARRAY_INDEX) > return -1; > - if (insert_new_key(tbl, &key, item_idx) =3D=3D > + if (insert_new_flow(tbl, &key, item_idx) =3D=3D > INVALID_ARRAY_INDEX) { > - /* > - * fail to insert a new key, so > - * delete the inserted item > - */ > + /* Fail to insert a new flow. */ > delete_item(tbl, item_idx, INVALID_ARRAY_INDEX); > return -1; > } > return 0; > } >=20 > - /* traverse all packets in the item group to find one to merge */ > - cur_idx =3D tbl->keys[i].start_index; > + /* > + * Check all packets in the flow and try to find a neighbor for > + * the input packet. > + */ > + cur_idx =3D tbl->flows[i].start_index; > prev_idx =3D cur_idx; > do { > cmp =3D check_seq_option(&(tbl->items[cur_idx]), tcp_hdr, > - pkt->l4_len, tcp_dl, ip_id, sent_seq); > + sent_seq, ip_id, pkt->l4_len, tcp_dl, 0, > + is_atomic); > if (cmp) { > if (merge_two_tcp4_packets(&(tbl->items[cur_idx]), > - pkt, ip_id, > - sent_seq, cmp)) > + pkt, cmp, sent_seq, ip_id, 0)) > return 1; > /* > - * fail to merge two packets since the packet > - * length will be greater than the max value. > - * So insert the packet into the item group. > + * Fail to merge the two packets, as the packet > + * length is greater than the max value. Store > + * the packet into the flow. > */ > - if (insert_new_item(tbl, pkt, ip_id, sent_seq, > - prev_idx, start_time) =3D=3D > + if (insert_new_item(tbl, pkt, start_time, prev_idx, > + sent_seq, ip_id, > + is_atomic) =3D=3D > INVALID_ARRAY_INDEX) > return -1; > return 0; > @@ -429,12 +323,9 @@ gro_tcp4_reassemble(struct rte_mbuf *pkt, > cur_idx =3D tbl->items[cur_idx].next_pkt_idx; > } while (cur_idx !=3D INVALID_ARRAY_INDEX); >=20 > - /* > - * can't find a packet in the item group to merge, > - * so insert the packet into the item group. > - */ > - if (insert_new_item(tbl, pkt, ip_id, sent_seq, prev_idx, > - start_time) =3D=3D INVALID_ARRAY_INDEX) > + /* Fail to find a neighbor, so store the packet into the flow. */ > + if (insert_new_item(tbl, pkt, start_time, prev_idx, sent_seq, > + ip_id, is_atomic) =3D=3D INVALID_ARRAY_INDEX) > return -1; >=20 > return 0; > @@ -446,46 +337,35 @@ gro_tcp4_tbl_timeout_flush(struct gro_tcp4_tbl > *tbl, > struct rte_mbuf **out, > uint16_t nb_out) > { > - uint16_t k =3D 0; > + uint32_t max_flow_num =3D tbl->max_flow_num; > uint32_t i, j; > - uint32_t max_key_num =3D tbl->max_key_num; > + uint16_t k =3D 0; >=20 > - for (i =3D 0; i < max_key_num; i++) { > - /* all keys have been checked, return immediately */ > - if (tbl->key_num =3D=3D 0) > + for (i =3D 0; i < max_flow_num; i++) { > + if (unlikely(tbl->flow_num =3D=3D 0)) > return k; >=20 > - j =3D tbl->keys[i].start_index; > + j =3D tbl->flows[i].start_index; > while (j !=3D INVALID_ARRAY_INDEX) { > if (tbl->items[j].start_time <=3D flush_timestamp) { > out[k++] =3D tbl->items[j].firstseg; > if (tbl->items[j].nb_merged > 1) > update_header(&(tbl->items[j])); > /* > - * delete the item and get > - * the next packet index > + * Delete the packet and get the next > + * packet in the flow. > */ > - j =3D delete_item(tbl, j, > - INVALID_ARRAY_INDEX); > + j =3D delete_item(tbl, j, INVALID_ARRAY_INDEX); > + tbl->flows[i].start_index =3D j; > + if (j =3D=3D INVALID_ARRAY_INDEX) > + tbl->flow_num--; >=20 > - /* > - * delete the key as all of > - * packets are flushed > - */ > - if (j =3D=3D INVALID_ARRAY_INDEX) { > - tbl->keys[i].start_index =3D > - INVALID_ARRAY_INDEX; > - tbl->key_num--; > - } else > - /* update start_index of the key */ > - tbl->keys[i].start_index =3D j; > - > - if (k =3D=3D nb_out) > + if (unlikely(k =3D=3D nb_out)) > return k; > } else > /* > - * left packets of this key won't be > - * timeout, so go to check other keys. > + * The left packets in this flow won't be > + * timeout. Go to check other flows. > */ > break; > } > diff --git a/lib/librte_gro/gro_tcp4.h b/lib/librte_gro/gro_tcp4.h > index 0a81716..66d6ce9 100644 > --- a/lib/librte_gro/gro_tcp4.h > +++ b/lib/librte_gro/gro_tcp4.h > @@ -33,17 +33,20 @@ > #ifndef _GRO_TCP4_H_ > #define _GRO_TCP4_H_ >=20 > +#include > +#include > + > #define INVALID_ARRAY_INDEX 0xffffffffUL > #define GRO_TCP4_TBL_MAX_ITEM_NUM (1024UL * 1024UL) >=20 > /* > - * the max L3 length of a TCP/IPv4 packet. The L3 length > - * is the sum of ipv4 header, tcp header and L4 payload. > + * The max length of a IPv4 packet, which includes the length of the L3 > + * header, the L4 header and the data payload. > */ > -#define TCP4_MAX_L3_LENGTH UINT16_MAX > +#define MAX_IPV4_PKT_LENGTH UINT16_MAX >=20 > -/* criteria of mergeing packets */ > -struct tcp4_key { > +/* Header fields representing a TCP/IPv4 flow */ > +struct tcp4_flow_key { > struct ether_addr eth_saddr; > struct ether_addr eth_daddr; > uint32_t ip_src_addr; > @@ -54,77 +57,76 @@ struct tcp4_key { > uint16_t dst_port; > }; >=20 > -struct gro_tcp4_key { > - struct tcp4_key key; > +struct gro_tcp4_flow { > + struct tcp4_flow_key key; > /* > - * the index of the first packet in the item group. > - * If the value is INVALID_ARRAY_INDEX, it means > - * the key is empty. > + * The index of the first packet in the flow. > + * INVALID_ARRAY_INDEX indicates an empty flow. > */ > uint32_t start_index; > }; >=20 > struct gro_tcp4_item { > /* > - * first segment of the packet. If the value > + * The first MBUF segment of the packet. If the value > * is NULL, it means the item is empty. > */ > struct rte_mbuf *firstseg; > - /* last segment of the packet */ > + /* The last MBUF segment of the packet */ > struct rte_mbuf *lastseg; > /* > - * the time when the first packet is inserted > - * into the table. If a packet in the table is > - * merged with an incoming packet, this value > - * won't be updated. We set this value only > - * when the first packet is inserted into the > - * table. > + * The time when the first packet is inserted into the table. > + * This value won't be updated, even if the packet is merged > + * with other packets. > */ > uint64_t start_time; > /* > - * we use next_pkt_idx to chain the packets that > - * have same key value but can't be merged together. > + * next_pkt_idx is used to chain the packets that > + * are in the same flow but can't be merged together > + * (e.g. caused by packet reordering). > */ > uint32_t next_pkt_idx; > - /* the sequence number of the packet */ > + /* TCP sequence number of the packet */ > uint32_t sent_seq; > - /* the IP ID of the packet */ > + /* IPv4 ID of the packet */ > uint16_t ip_id; > - /* the number of merged packets */ > + /* The number of merged packets */ > uint16_t nb_merged; > + /* Indicate if IPv4 ID can be ignored */ > + uint8_t is_atomic; > }; >=20 > /* > - * TCP/IPv4 reassembly table structure. > + * TCP/IPv4 reassembly table structure > */ > struct gro_tcp4_tbl { > /* item array */ > struct gro_tcp4_item *items; > - /* key array */ > - struct gro_tcp4_key *keys; > + /* flow array */ > + struct gro_tcp4_flow *flows; > /* current item number */ > uint32_t item_num; > - /* current key num */ > - uint32_t key_num; > + /* current flow num */ > + uint32_t flow_num; > /* item array size */ > uint32_t max_item_num; > - /* key array size */ > - uint32_t max_key_num; > + /* flow array size */ > + uint32_t max_flow_num; > }; >=20 > /** > * This function creates a TCP/IPv4 reassembly table. > * > * @param socket_id > - * socket index for allocating TCP/IPv4 reassemble table > + * Socket index for allocating the TCP/IPv4 reassemble table > * @param max_flow_num > - * the maximum number of flows in the TCP/IPv4 GRO table > + * The maximum number of flows in the TCP/IPv4 GRO table > * @param max_item_per_flow > - * the maximum packet number per flow. > + * The maximum number of packets per flow > * > * @return > - * if create successfully, return a pointer which points to the > - * created TCP/IPv4 GRO table. Otherwise, return NULL. > + * - Return the table pointer on success. > + * - Return NULL on failure. > */ > void *gro_tcp4_tbl_create(uint16_t socket_id, > uint16_t max_flow_num, > @@ -134,62 +136,56 @@ void *gro_tcp4_tbl_create(uint16_t socket_id, > * This function destroys a TCP/IPv4 reassembly table. > * > * @param tbl > - * a pointer points to the TCP/IPv4 reassembly table. > + * Pointer pointing to the TCP/IPv4 reassembly table. > */ > void gro_tcp4_tbl_destroy(void *tbl); >=20 > /** > - * This function searches for a packet in the TCP/IPv4 reassembly table > - * to merge with the inputted one. To merge two packets is to chain them > - * together and update packet headers. Packets, whose SYN, FIN, RST, PSH > - * CWR, ECE or URG bit is set, are returned immediately. Packets which > - * only have packet headers (i.e. without data) are also returned > - * immediately. Otherwise, the packet is either merged, or inserted into > - * the table. Besides, if there is no available space to insert the > - * packet, this function returns immediately too. > + * This function merges a TCP/IPv4 packet. It doesn't process the packet= , > + * which has SYN, FIN, RST, PSH, CWR, ECE or URG set, or doesn't have > + * payload. > * > - * This function assumes the inputted packet is with correct IPv4 and > - * TCP checksums. And if two packets are merged, it won't re-calculate > - * IPv4 and TCP checksums. Besides, if the inputted packet is IP > - * fragmented, it assumes the packet is complete (with TCP header). > + * This function doesn't check if the packet has correct checksums and > + * doesn't re-calculate checksums for the merged packet. Additionally, > + * it assumes the packets are complete (i.e., MF=3D=3D0 && frag_off=3D= =3D0), > + * when IP fragmentation is possible (i.e., DF=3D=3D0). It returns the > + * packet, if the packet has invalid parameters (e.g. SYN bit is set) > + * or there is no available space in the table. > * > * @param pkt > - * packet to reassemble. > + * Packet to reassemble > * @param tbl > - * a pointer that points to a TCP/IPv4 reassembly table. > + * Pointer pointing to the TCP/IPv4 reassembly table > * @start_time > - * the start time that the packet is inserted into the table > + * The time when the packet is inserted into the table > * > * @return > - * if the packet doesn't have data, or SYN, FIN, RST, PSH, CWR, ECE > - * or URG bit is set, or there is no available space in the table to > - * insert a new item or a new key, return a negative value. If the > - * packet is merged successfully, return an positive value. If the > - * packet is inserted into the table, return 0. > + * - Return a positive value if the packet is merged. > + * - Return zero if the packet isn't merged but stored in the table. > + * - Return a negative value for invalid parameters or no available > + * space in the table. > */ > int32_t gro_tcp4_reassemble(struct rte_mbuf *pkt, > struct gro_tcp4_tbl *tbl, > uint64_t start_time); >=20 > /** > - * This function flushes timeout packets in a TCP/IPv4 reassembly table > - * to applications, and without updating checksums for merged packets. > - * The max number of flushed timeout packets is the element number of > - * the array which is used to keep flushed packets. > + * This function flushes timeout packets in a TCP/IPv4 reassembly table, > + * and without updating checksums. > * > * @param tbl > - * a pointer that points to a TCP GRO table. > + * TCP/IPv4 reassembly table pointer > * @param flush_timestamp > - * this function flushes packets which are inserted into the table > - * before or at the flush_timestamp. > + * Flush packets which are inserted into the table before or at the > + * flush_timestamp. > * @param out > - * pointer array which is used to keep flushed packets. > + * Pointer array used to keep flushed packets > * @param nb_out > - * the element number of out. It's also the max number of timeout > + * The element number in 'out'. It also determines the maximum number o= f > * packets that can be flushed finally. > * > * @return > - * the number of packets that are returned. > + * The number of flushed packets > */ > uint16_t gro_tcp4_tbl_timeout_flush(struct gro_tcp4_tbl *tbl, > uint64_t flush_timestamp, > @@ -201,10 +197,131 @@ uint16_t gro_tcp4_tbl_timeout_flush(struct > gro_tcp4_tbl *tbl, > * reassembly table. > * > * @param tbl > - * pointer points to a TCP/IPv4 reassembly table. > + * TCP/IPv4 reassembly table pointer > * > * @return > - * the number of packets in the table > + * The number of packets in the table > */ > uint32_t gro_tcp4_tbl_pkt_count(void *tbl); > + > +/* > + * Check if two TCP/IPv4 packets belong to the same flow. > + */ > +static inline int > +is_same_tcp4_flow(struct tcp4_flow_key k1, struct tcp4_flow_key k2) > +{ > + return (is_same_ether_addr(&k1.eth_saddr, &k2.eth_saddr) && > + is_same_ether_addr(&k1.eth_daddr, &k2.eth_daddr) && > + (k1.ip_src_addr =3D=3D k2.ip_src_addr) && > + (k1.ip_dst_addr =3D=3D k2.ip_dst_addr) && > + (k1.recv_ack =3D=3D k2.recv_ack) && > + (k1.src_port =3D=3D k2.src_port) && > + (k1.dst_port =3D=3D k2.dst_port)); > +} > + > +/* > + * Check if two TCP/IPv4 packets are neighbors. > + */ > +static inline int > +check_seq_option(struct gro_tcp4_item *item, > + struct tcp_hdr *tcph, > + uint32_t sent_seq, > + uint16_t ip_id, > + uint16_t tcp_hl, > + uint16_t tcp_dl, > + uint16_t l2_offset, > + uint8_t is_atomic) > +{ > + struct rte_mbuf *pkt_orig =3D item->firstseg; > + struct ipv4_hdr *iph_orig; > + struct tcp_hdr *tcph_orig; > + uint16_t len, l4_len_orig; > + > + iph_orig =3D (struct ipv4_hdr *)(rte_pktmbuf_mtod(pkt_orig, char *) + > + l2_offset + pkt_orig->l2_len); > + tcph_orig =3D (struct tcp_hdr *)((char *)iph_orig + pkt_orig->l3_len); > + l4_len_orig =3D pkt_orig->l4_len; > + > + /* Check if TCP option fields equal */ > + len =3D RTE_MAX(tcp_hl, l4_len_orig) - sizeof(struct tcp_hdr); > + if ((tcp_hl !=3D l4_len_orig) || ((len > 0) && > + (memcmp(tcph + 1, tcph_orig + 1, > + len) !=3D 0))) > + return 0; > + > + /* Don't merge packets whose DF bits are different */ > + if (unlikely(item->is_atomic ^ is_atomic)) > + return 0; > + > + /* Check if the two packets are neighbors */ > + len =3D pkt_orig->pkt_len - l2_offset - pkt_orig->l2_len - > + pkt_orig->l3_len - l4_len_orig; > + if ((sent_seq =3D=3D item->sent_seq + len) && (is_atomic || > + (ip_id =3D=3D item->ip_id + item->nb_merged))) > + /* Append the new packet */ > + return 1; > + else if ((sent_seq + tcp_dl =3D=3D item->sent_seq) && (is_atomic || > + (ip_id + 1 =3D=3D item->ip_id))) > + /* Pre-pend the new packet */ > + return -1; > + > + return 0; > +} > + > +/* > + * Merge two TCP/IPv4 packets without updating checksums. > + * If cmp is larger than 0, append the new packet to the > + * original packet. Otherwise, pre-pend the new packet to > + * the original packet. > + */ > +static inline int > +merge_two_tcp4_packets(struct gro_tcp4_item *item, > + struct rte_mbuf *pkt, > + int cmp, > + uint32_t sent_seq, > + uint16_t ip_id, > + uint16_t l2_offset) > +{ > + struct rte_mbuf *pkt_head, *pkt_tail, *lastseg; > + uint16_t hdr_len, l2_len; > + > + if (cmp > 0) { > + pkt_head =3D item->firstseg; > + pkt_tail =3D pkt; > + } else { > + pkt_head =3D pkt; > + pkt_tail =3D item->firstseg; > + } > + > + /* Check if the IPv4 packet length is greater than the max value */ > + hdr_len =3D l2_offset + pkt_head->l2_len + pkt_head->l3_len + > + pkt_head->l4_len; > + l2_len =3D l2_offset > 0 ? pkt_head->outer_l2_len : pkt_head->l2_len; > + if (unlikely(pkt_head->pkt_len - l2_len + pkt_tail->pkt_len - hdr_len > > + MAX_IPV4_PKT_LENGTH)) > + return 0; > + > + /* Remove the packet header */ > + rte_pktmbuf_adj(pkt_tail, hdr_len); > + > + /* Chain two packets together */ > + if (cmp > 0) { > + item->lastseg->next =3D pkt; > + item->lastseg =3D rte_pktmbuf_lastseg(pkt); > + } else { > + lastseg =3D rte_pktmbuf_lastseg(pkt); > + lastseg->next =3D item->firstseg; > + item->firstseg =3D pkt; > + /* Update sent_seq and ip_id */ > + item->sent_seq =3D sent_seq; > + item->ip_id =3D ip_id; > + } > + item->nb_merged++; > + > + /* Update MBUF metadata for the merged packet */ > + pkt_head->nb_segs +=3D pkt_tail->nb_segs; > + pkt_head->pkt_len +=3D pkt_tail->pkt_len; > + > + return 1; > +} > #endif > diff --git a/lib/librte_gro/rte_gro.c b/lib/librte_gro/rte_gro.c > index 7853246..b3931a8 100644 > --- a/lib/librte_gro/rte_gro.c > +++ b/lib/librte_gro/rte_gro.c > @@ -51,11 +51,14 @@ static gro_tbl_destroy_fn > tbl_destroy_fn[RTE_GRO_TYPE_MAX_NUM] =3D { > static gro_tbl_pkt_count_fn tbl_pkt_count_fn[RTE_GRO_TYPE_MAX_NUM] =3D > { > gro_tcp4_tbl_pkt_count, NULL}; >=20 > +#define IS_IPV4_TCP_PKT(ptype) (RTE_ETH_IS_IPV4_HDR(ptype) && \ > + ((ptype & RTE_PTYPE_L4_TCP) =3D=3D RTE_PTYPE_L4_TCP)) > + > /* > - * GRO context structure, which is used to merge packets. It keeps > - * many reassembly tables of desired GRO types. Applications need to > - * create GRO context objects before using rte_gro_reassemble to > - * perform GRO. > + * GRO context structure. It keeps the table structures, which are > + * used to merge packets, for different GRO types. Before using > + * rte_gro_reassemble(), applications need to create the GRO context > + * first. > */ > struct gro_ctx { > /* GRO types to perform */ > @@ -93,7 +96,7 @@ rte_gro_ctx_create(const struct rte_gro_param *param) > param->max_flow_num, > param->max_item_per_flow); > if (gro_ctx->tbls[i] =3D=3D NULL) { > - /* destroy all created tables */ > + /* Destroy all created tables */ > gro_ctx->gro_types =3D gro_types; > rte_gro_ctx_destroy(gro_ctx); > return NULL; > @@ -113,8 +116,6 @@ rte_gro_ctx_destroy(void *ctx) > uint64_t gro_type_flag; > uint8_t i; >=20 > - if (gro_ctx =3D=3D NULL) > - return; > for (i =3D 0; i < RTE_GRO_TYPE_MAX_NUM; i++) { > gro_type_flag =3D 1ULL << i; > if ((gro_ctx->gro_types & gro_type_flag) =3D=3D 0) > @@ -131,62 +132,54 @@ rte_gro_reassemble_burst(struct rte_mbuf **pkts, > uint16_t nb_pkts, > const struct rte_gro_param *param) > { > - uint16_t i; > - uint16_t nb_after_gro =3D nb_pkts; > - uint32_t item_num; > - > - /* allocate a reassembly table for TCP/IPv4 GRO */ > + /* Allocate a reassembly table for TCP/IPv4 GRO */ > struct gro_tcp4_tbl tcp_tbl; > - struct gro_tcp4_key tcp_keys[RTE_GRO_MAX_BURST_ITEM_NUM]; > + struct gro_tcp4_flow tcp_flows[RTE_GRO_MAX_BURST_ITEM_NUM]; > struct gro_tcp4_item tcp_items[RTE_GRO_MAX_BURST_ITEM_NUM] =3D > {{0} }; >=20 > struct rte_mbuf *unprocess_pkts[nb_pkts]; > - uint16_t unprocess_num =3D 0; > + uint32_t item_num; > int32_t ret; > - uint64_t current_time; > + uint16_t i, unprocess_num =3D 0, nb_after_gro =3D nb_pkts; >=20 > - if ((param->gro_types & RTE_GRO_TCP_IPV4) =3D=3D 0) > + if (unlikely((param->gro_types & RTE_GRO_TCP_IPV4) =3D=3D 0)) > return nb_pkts; >=20 > - /* get the actual number of packets */ > + /* Get the maximum number of packets */ > item_num =3D RTE_MIN(nb_pkts, (param->max_flow_num * > - param->max_item_per_flow)); > + param->max_item_per_flow)); > item_num =3D RTE_MIN(item_num, RTE_GRO_MAX_BURST_ITEM_NUM); >=20 > for (i =3D 0; i < item_num; i++) > - tcp_keys[i].start_index =3D INVALID_ARRAY_INDEX; > + tcp_flows[i].start_index =3D INVALID_ARRAY_INDEX; >=20 > - tcp_tbl.keys =3D tcp_keys; > + tcp_tbl.flows =3D tcp_flows; > tcp_tbl.items =3D tcp_items; > - tcp_tbl.key_num =3D 0; > + tcp_tbl.flow_num =3D 0; > tcp_tbl.item_num =3D 0; > - tcp_tbl.max_key_num =3D item_num; > + tcp_tbl.max_flow_num =3D item_num; > tcp_tbl.max_item_num =3D item_num; >=20 > - current_time =3D rte_rdtsc(); > - > for (i =3D 0; i < nb_pkts; i++) { > - if ((pkts[i]->packet_type & (RTE_PTYPE_L3_IPV4 | > - RTE_PTYPE_L4_TCP)) =3D=3D > - (RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_TCP)) { > - ret =3D gro_tcp4_reassemble(pkts[i], > - &tcp_tbl, > - current_time); > + if (IS_IPV4_TCP_PKT(pkts[i]->packet_type)) { > + /* > + * The timestamp is ignored, since all packets > + * will be flushed from the tables. > + */ > + ret =3D gro_tcp4_reassemble(pkts[i], &tcp_tbl, 0); > if (ret > 0) > - /* merge successfully */ > + /* Merge successfully */ > nb_after_gro--; > - else if (ret < 0) { > - unprocess_pkts[unprocess_num++] =3D > - pkts[i]; > - } > + else if (ret < 0) > + unprocess_pkts[unprocess_num++] =3D pkts[i]; > } else > unprocess_pkts[unprocess_num++] =3D pkts[i]; > } >=20 > - /* re-arrange GROed packets */ > if (nb_after_gro < nb_pkts) { > - i =3D gro_tcp4_tbl_timeout_flush(&tcp_tbl, current_time, > - pkts, nb_pkts); > + /* Flush all packets from the tables */ > + i =3D gro_tcp4_tbl_timeout_flush(&tcp_tbl, 0, pkts, nb_pkts); > + /* Copy unprocessed packets */ > if (unprocess_num > 0) { > memcpy(&pkts[i], unprocess_pkts, > sizeof(struct rte_mbuf *) * > @@ -202,31 +195,28 @@ rte_gro_reassemble(struct rte_mbuf **pkts, > uint16_t nb_pkts, > void *ctx) > { > - uint16_t i, unprocess_num =3D 0; > struct rte_mbuf *unprocess_pkts[nb_pkts]; > struct gro_ctx *gro_ctx =3D ctx; > + void *tcp_tbl; > uint64_t current_time; > + uint16_t i, unprocess_num =3D 0; >=20 > - if ((gro_ctx->gro_types & RTE_GRO_TCP_IPV4) =3D=3D 0) > + if (unlikely((gro_ctx->gro_types & RTE_GRO_TCP_IPV4) =3D=3D 0)) > return nb_pkts; >=20 > + tcp_tbl =3D gro_ctx->tbls[RTE_GRO_TCP_IPV4_INDEX]; > current_time =3D rte_rdtsc(); >=20 > for (i =3D 0; i < nb_pkts; i++) { > - if ((pkts[i]->packet_type & (RTE_PTYPE_L3_IPV4 | > - RTE_PTYPE_L4_TCP)) =3D=3D > - (RTE_PTYPE_L3_IPV4 | RTE_PTYPE_L4_TCP)) { > - if (gro_tcp4_reassemble(pkts[i], > - gro_ctx->tbls > - [RTE_GRO_TCP_IPV4_INDEX], > + if (IS_IPV4_TCP_PKT(pkts[i]->packet_type)) { > + if (gro_tcp4_reassemble(pkts[i], tcp_tbl, > current_time) < 0) > unprocess_pkts[unprocess_num++] =3D pkts[i]; > } else > unprocess_pkts[unprocess_num++] =3D pkts[i]; > } > if (unprocess_num > 0) { > - memcpy(pkts, unprocess_pkts, > - sizeof(struct rte_mbuf *) * > + memcpy(pkts, unprocess_pkts, sizeof(struct rte_mbuf *) * > unprocess_num); > } >=20 > @@ -252,6 +242,7 @@ rte_gro_timeout_flush(void *ctx, > flush_timestamp, > out, max_nb_out); > } > + > return 0; > } >=20 > @@ -262,7 +253,7 @@ rte_gro_get_pkt_count(void *ctx) > gro_tbl_pkt_count_fn pkt_count_fn; > uint64_t item_num =3D 0; > uint64_t gro_type_flag; > - uint8_t i; > + uint8_t gro_type_num =3D RTE_GRO_TYPE_SUPPORT_NUM, i; >=20 > for (i =3D 0; i < RTE_GRO_TYPE_MAX_NUM; i++) { > gro_type_flag =3D 1ULL << i; > @@ -270,9 +261,12 @@ rte_gro_get_pkt_count(void *ctx) > continue; >=20 > pkt_count_fn =3D tbl_pkt_count_fn[i]; > - if (pkt_count_fn =3D=3D NULL) > - continue; > - item_num +=3D pkt_count_fn(gro_ctx->tbls[i]); > + if (pkt_count_fn) { > + item_num +=3D pkt_count_fn(gro_ctx->tbls[i]); > + if (--gro_type_num =3D=3D 0) > + break; > + } > } > + > return item_num; > } > diff --git a/lib/librte_gro/rte_gro.h b/lib/librte_gro/rte_gro.h > index d57e0c5..36a1e60 100644 > --- a/lib/librte_gro/rte_gro.h > +++ b/lib/librte_gro/rte_gro.h > @@ -59,8 +59,8 @@ extern "C" { > /**< TCP/IPv4 GRO flag */ >=20 > /** > - * A structure which is used to create GRO context objects or tell > - * rte_gro_reassemble_burst() what reassembly rules are demanded. > + * Structure used to create GRO context objects or used to pass > + * application-determined parameters to rte_gro_reassemble_burst(). > */ > struct rte_gro_param { > uint64_t gro_types; > @@ -106,26 +106,23 @@ void rte_gro_ctx_destroy(void *ctx); >=20 > /** > * This is one of the main reassembly APIs, which merges numbers of > - * packets at a time. It assumes that all inputted packets are with > - * correct checksums. That is, applications should guarantee all > - * inputted packets are correct. Besides, it doesn't re-calculate > - * checksums for merged packets. If inputted packets are IP fragmented, > - * this function assumes them are complete (i.e. with L4 header). After > - * finishing processing, it returns all GROed packets to applications > - * immediately. > + * packets at a time. It doesn't check if input packets have correct > + * checksums and doesn't re-calculate checksums for merged packets. > + * It assumes the packets are complete (i.e., MF=3D=3D0 && frag_off=3D= =3D0), > + * when IP fragmentation is possible (i.e., DF=3D=3D1). The GROed packet= s > + * are returned as soon as the function finishes. > * > * @param pkts > - * a pointer array which points to the packets to reassemble. Besides, > - * it keeps mbuf addresses for the GROed packets. > + * Pointer array pointing to the packets to reassemble. Besides, it > + * keeps MBUF addresses for the GROed packets. > * @param nb_pkts > - * the number of packets to reassemble. > + * The number of packets to reassemble > * @param param > - * applications use it to tell rte_gro_reassemble_burst() what rules > - * are demanded. > + * Application-determined parameters for reassembling packets. > * > * @return > - * the number of packets after been GROed. If no packets are merged, > - * the returned value is nb_pkts. > + * The number of packets after been GROed. If no packets are merged, > + * the return value is equals to nb_pkts. > */ > uint16_t rte_gro_reassemble_burst(struct rte_mbuf **pkts, > uint16_t nb_pkts, > @@ -135,32 +132,28 @@ uint16_t rte_gro_reassemble_burst(struct rte_mbuf > **pkts, > * @warning > * @b EXPERIMENTAL: this API may change without prior notice > * > - * Reassembly function, which tries to merge inputted packets with > - * the packets in the reassembly tables of a given GRO context. This > - * function assumes all inputted packets are with correct checksums. > - * And it won't update checksums if two packets are merged. Besides, > - * if inputted packets are IP fragmented, this function assumes they > - * are complete packets (i.e. with L4 header). > + * Reassembly function, which tries to merge input packets with the > + * existed packets in the reassembly tables of a given GRO context. > + * It doesn't check if input packets have correct checksums and doesn't > + * re-calculate checksums for merged packets. Additionally, it assumes > + * the packets are complete (i.e., MF=3D=3D0 && frag_off=3D=3D0), when I= P > + * fragmentation is possible (i.e., DF=3D=3D1). > * > - * If the inputted packets don't have data or are with unsupported GRO > - * types etc., they won't be processed and are returned to applications. > - * Otherwise, the inputted packets are either merged or inserted into > - * the table. If applications want get packets in the table, they need > - * to call flush API. > + * If the input packets have invalid parameters (e.g. no data payload, > + * unsupported GRO types), they are returned to applications. Otherwise, > + * they are either merged or inserted into the table. Applications need > + * to flush packets from the tables by flush API, if they want to get th= e > + * GROed packets. > * > * @param pkts > - * packet to reassemble. Besides, after this function finishes, it > - * keeps the unprocessed packets (e.g. without data or unsupported > - * GRO types). > + * Packets to reassemble. It's also used to store the unprocessed packe= ts. > * @param nb_pkts > - * the number of packets to reassemble. > + * The number of packets to reassemble > * @param ctx > - * a pointer points to a GRO context object. > + * GRO context object pointer > * > * @return > - * return the number of unprocessed packets (e.g. without data or > - * unsupported GRO types). If all packets are processed (merged or > - * inserted into the table), return 0. > + * The number of unprocessed packets. > */ > uint16_t rte_gro_reassemble(struct rte_mbuf **pkts, > uint16_t nb_pkts, > @@ -170,29 +163,28 @@ uint16_t rte_gro_reassemble(struct rte_mbuf > **pkts, > * @warning > * @b EXPERIMENTAL: this API may change without prior notice > * > - * This function flushes the timeout packets from reassembly tables of > - * desired GRO types. The max number of flushed timeout packets is the > - * element number of the array which is used to keep the flushed packets= . > + * This function flushes the timeout packets from the reassembly tables > + * of desired GRO types. The max number of flushed packets is the > + * element number of 'out'. > * > - * Besides, this function won't re-calculate checksums for merged > - * packets in the tables. That is, the returned packets may be with > - * wrong checksums. > + * Additionally, the flushed packets may have incorrect checksums, since > + * this function doesn't re-calculate checksums for merged packets. > * > * @param ctx > - * a pointer points to a GRO context object. > + * GRO context object pointer. > * @param timeout_cycles > - * max TTL for packets in reassembly tables, measured in nanosecond. > + * The max TTL for packets in reassembly tables, measured in nanosecond= . > * @param gro_types > - * this function only flushes packets which belong to the GRO types > - * specified by gro_types. > + * This function flushes packets whose GRO types are specified by > + * gro_types. > * @param out > - * a pointer array that is used to keep flushed timeout packets. > + * Pointer array used to keep flushed packets. > * @param max_nb_out > - * the element number of out. It's also the max number of timeout > + * The element number of 'out'. It's also the max number of timeout > * packets that can be flushed finally. > * > * @return > - * the number of flushed packets. If no packets are flushed, return 0. > + * The number of flushed packets. > */ > uint16_t rte_gro_timeout_flush(void *ctx, > uint64_t timeout_cycles, > @@ -208,10 +200,10 @@ uint16_t rte_gro_timeout_flush(void *ctx, > * of a given GRO context. > * > * @param ctx > - * pointer points to a GRO context object. > + * GRO context object pointer. > * > * @return > - * the number of packets in all reassembly tables. > + * The number of packets in the tables. > */ > uint64_t rte_gro_get_pkt_count(void *ctx); >=20 > -- > 2.7.4 Reviewed-by: Junjie Chen Thanks