Re: [dpdk-users] Unable to merge packets using GRO feature

["Hu, Jiayu" <jiayu.hu@intel.com>]

Hi,

Sorry to reply so late.

When test the GRO feature, we use the following experimental setup:

==================================================================
Machine A:
1) two physical FVL ports are physically linked. One is assigned to DPDK (called p0)
and another is to kernel (called p1).
2) Launch testpmd with a vhost port (called vp0) and p0. One thing to notice:
don't use vector TX function for p0, since we will enable csum offloading for p0.
3) Launch a VM with a virtio-net port (called vp1) and "mrg_rxbuf=on".

===================================================
P1: enable TSO and run iperf client
vp1: disable kernel GRO and run iperf server
Testpmd: comment out two "ether_addr_copy()" in csumonly.c
Testpmd commands:
> set fwd csum
>csum set ip hw p0
>csum set tcp hw p0
>csum set tcp hw vp0
>gro on p0
>start

If this setup still doesn't work for you, I can send you our detailed test steps. Additionally,
I will double-check if the GRO feature works correctly in a similar experimental topology
you used. If you meet any issues, please let me know.

Thanks,
Jiayu

> -----Original Message-----
> From: Wisam Monther [mailto:wisamm@mellanox.com]
> Sent: Monday, August 28, 2017 3:37 PM
> To: Hu, Jiayu <jiayu.hu@intel.com>
> Cc: users@dpdk.org; Raslan Darawsheh <rasland@mellanox.com>; Shahaf
> Shuler <shahafs@mellanox.com>
> Subject: RE: Unable to merge packets using GRO feature
> 
> Hi Jiayu,
> 
> I'm sorry for bothering you, but could you conform that the feature is
> working probably
> Because, what I ever  did, I couldn't get the merged packets.
> 
> BRs,
> Wisam Jaddo
> 
> -----Original Message-----
> From: Wisam Monther
> Sent: Thursday, August 24, 2017 9:15 AM
> To: 'Hu, Jiayu'
> Cc: users@dpdk.org; Raslan Darawsheh; Shahaf Shuler
> Subject: RE: Unable to merge packets using GRO feature
> 
> Hi,
> 
> I'm using Mellanox NICs, and it is supporting parse packet types.
> 
> Best regards,
> Wisam Jaddo
> 
> -----Original Message-----
> From: Hu, Jiayu [mailto:jiayu.hu@intel.com]
> Sent: Thursday, August 24, 2017 8:47 AM
> To: Wisam Monther
> Cc: users@dpdk.org; Raslan Darawsheh; Shahaf Shuler
> Subject: RE: Unable to merge packets using GRO feature
> 
> Hi,
> 
> Can you tell me what's the NIC type of the GRO-enabled port?
> 
> Since GRO library uses mbuf->packet_type to parse packet headers,
> applications need to fill this value before calling GRO reassembly APIs.
> Otherwise, the GRO can't work correctly.
> 
> In csum forwarding engine of testpmd, packet_type is filled by NIC drivers.
> The csum forwarding engine won't set this value. So if your NIC doesn’t
> support to parse packet types, the value of packet_type is 0 and GRO can't
> work correctly.
> 
> BRs,
> Jiayu
> 
> > -----Original Message-----
> > From: Wisam Monther [mailto:wisamm@mellanox.com]
> > Sent: Tuesday, August 22, 2017 9:25 PM
> > To: Hu, Jiayu <jiayu.hu@intel.com>
> > Cc: users@dpdk.org; Raslan Darawsheh <rasland@mellanox.com>; Shahaf
> > Shuler <shahafs@mellanox.com>
> > Subject: RE: Unable to merge packets using GRO feature
> >
> > Yes it is,
> > The fragmented packets comes from port1 / NIC b from machine A to port
> > 1 in NIC A for machine b So it's received on the port '1', which is
> > configured gro active on this port.
> >
> > Best regards,
> > Wisam Jaddo
> >
> > -----Original Message-----
> > From: Hu, Jiayu [mailto:jiayu.hu@intel.com]
> > Sent: Tuesday, August 22, 2017 4:21 PM
> > To: Wisam Monther
> > Cc: users@dpdk.org; Raslan Darawsheh; Shahaf Shuler
> > Subject: RE: Unable to merge packets using GRO feature
> >
> > Hi,
> >
> > > -----Original Message-----
> > > From: Wisam Monther [mailto:wisamm@mellanox.com]
> > > Sent: Tuesday, August 22, 2017 7:07 PM
> > > To: Hu, Jiayu <jiayu.hu@intel.com>
> > > Cc: users@dpdk.org; Raslan Darawsheh <rasland@mellanox.com>;
> Shahaf
> > > Shuler <shahafs@mellanox.com>
> > > Subject: RE: Unable to merge packets using GRO feature
> > >
> > > Hey Jiayu,
> > >
> > > Thank you for your reply.
> > > I tried what you said with the csum at fwd mode.
> > > Even so the GRO didn't works fine.
> > >
> > > I even tested with a new methodology.
> > > Two machines with two different nic for each.
> > > The methodology that I used to test it is described in the attached file.
> > >
> > > What I did from gro side:
> > > """
> > > testpmd>gro on 1
> >
> > Does the port number of NIC A in machine B is '1'? When you enable GRO
> > for port '1', Testpmd only tries to merge packets received from port '1'.
> >
> > BRs,
> > Jiayu
> >
> > > Testpmd>set fwd csum
> > > Testpmd>start
> > > """
> > > And the packet with correct dst mac.
> > >
> > > Best regards,
> > > Wisam Jaddo
> > > -----Original Message-----
> > > From: Jiayu Hu [mailto:jiayu.hu@intel.com]
> > > Sent: Monday, August 21, 2017 12:14 PM
> > > To: Wisam Monther
> > > Cc: Thomas Monjalon; users@dpdk.org; Raslan Darawsheh; Shahaf Shuler
> > > Subject: Re: Unable to merge packets using GRO feature
> > >
> > > Hi,
> > >
> > > On Mon, Aug 21, 2017 at 07:25:23AM +0000, Wisam Monther wrote:
> > > > Hello Guys,
> > > >
> > > >
> > > >
> > > > I hope this finds you well, I’m trying to test the GRO feature.
> > > > But I’m stuck with this scenario.
> > > >
> > > > As you know, GRO is only support TCP_IPV4 packet until now.
> > > >
> > > > So I’m trying to test the basic functionality of the feature, as following:
> > > >
> > > > Start testpmd:
> > > >
> > > > “””
> > > >
> > > > ./x86_64-native-linuxapp-gcc/build/app/test-pmd/testpmd -n 4  -w
> > > > 00:0a.0  -w
> > > > 00:09.0 --  --burst=64 --mbcache=512 --portmask 0xf -i  --txd=512
> > > > --rxd=512
> > > > --nb-cores=9  --rxq=2 --txq=2 --txqflags=0
> > > >
> > > > “””
> > > >
> > > >
> > > >
> > > > Then enable GRO at the two ports:
> > > >
> > > > “””
> > > >
> > > > Testpmd>gro on 0
> > > >
> > > > Testpmd>gro on 1
> > >
> > > When use GRO in testpmd, there are following things to notice:
> > >
> > > 1. In testpmd, GRO is supported by csum forwarding engine.
> > > Therefore, please use 'set fwd csum' to switch forwarding engine.
> > >
> > > 2. By default, csum forwarding engine compulsorily changes ethernet
> > > addresses. So please make sure that MAC addresses are correct.
> > >
> > > 3. When enable GRO for port0, csum forwarding engine will merge
> > > packets received from port0. If there are no packets from port1 to
> > > port0, you don't need to enable GRO for port1.
> > >
> > > 4. GRO library doesn't re-calculate checksums for merged packets. If
> > > you want merged packets have correct checksum, please select HW IP
> > > and HW TCP checksum calculation for the port which the merged
> > > packets are transmitted to in csum forwarding engine.
> > > This is because the merged packets are multi-segment mbufs, but csum
> > > forwarding engine doesn't support to calculate checksums for
> > > multi-segment mbufs in SW. So we need to select HW checksum
> offloading.
> > >
> > > e.g. If data flow is "packets -> port0 -> port1", commands used in
> testpmd:
> > > 	gro on port0
> > > 	set fwd csum
> > > 	csum set ip hw port1
> > > 	csum set tcp hw port1
> > >
> > >
> > > Besides, you need to make sure that your PMD driver doesn't use
> > > vector TX function, since vector function doesn't support checksum
> offloading.
> > >
> > > >
> > > > “””
> > > >
> > > >
> > > >
> > > > And trying to send TCP_IPV4 fragmented packet “packet with length
> > > > 1500 fragmented to three packets of 500”
> > > >
> > > > “””
> > > >
> > > > p=Ether(src=get_if_hwaddr('ens10'), dst=
> > > > '24:8A:07:88:26:6B')/IP()/TCP()
> > > >
> > > > p.add_payload('F'*(1500 - len(p)))
> > > >
> > > > frags=fragment(p,fragsize=500)
> > > >
> > > > for fragment in frags:
> > > >
> > > >      sendp(fragment, iface='ens10')
> > > >
> > > > “””
> > > >
> > > >
> > > >
> > > > But the testpmd forward the packets as it is, “ doesn’t do any merge”
> > > >
> > > >
> > > >
> > > > Tcpdump at the TG side,
> > > >
> > > > The sending fragmets using ens10:
> > > >
> > > > #tcpdump –I ens10 –vvven
> > > >
> > > > 15:45:29.083514 24:8a:07:88:26:5b > 24:8a:07:88:26:6b, ethertype
> > > > IPv4 (0x0800), length 538: (tos 0x0, ttl 64, id 1, offset 0, flags
> > > > [+], proto Options (0), length 524)
> > > >
> > > >     127.0.0.1 > 127.0.0.1:  ip-proto-0 504
> > > >
> > > > 15:45:29.115266 24:8a:07:88:26:5b > 24:8a:07:88:26:6b, ethertype
> > > > IPv4 (0x0800), length 538: (tos 0x0, ttl 64, id 1, offset 504,
> > > > flags [+], proto Options (0), length 524)
> > > >
> > > >     127.0.0.1 > 127.0.0.1: ip-proto-0
> > > >
> > > > 15:45:29.147258 24:8a:07:88:26:5b > 24:8a:07:88:26:6b, ethertype
> > > > IPv4 (0x0800), length 492: (tos 0x0, ttl 64, id 1, offset 1008,
> > > > flags [none], proto Options (0), length 478)
> > > >
> > > >     127.0.0.1 > 127.0.0.1: ip-proto-0
> > > >
> > > >
> > > >
> > > > #tcpdump -i ens9 –vvven  /// here will be received the forwarded
> > > > packets from
> > > > testpmd:
> > > >
> > > > 15:45:29.083996 24:8a:07:88:26:5b > 24:8a:07:88:26:6b, ethertype
> > > > IPv4 (0x0800), length 538: (tos 0x0, ttl 64, id 1, offset 0, flags
> > > > [+], proto Options (0), length 524)
> > > >
> > > >     127.0.0.1 > 127.0.0.1:  ip-proto-0 504
> > > >
> > > > 15:45:29.115425 24:8a:07:88:26:5b > 24:8a:07:88:26:6b, ethertype
> > > > IPv4 (0x0800), length 538: (tos 0x0, ttl 64, id 1, offset 504,
> > > > flags [+], proto Options (0), length 524)
> > > >
> > > >     127.0.0.1 > 127.0.0.1: ip-proto-0
> > > >
> > > > 15:45:29.147492 24:8a:07:88:26:5b > 24:8a:07:88:26:6b, ethertype
> > > > IPv4 (0x0800), length 492: (tos 0x0, ttl 64, id 1, offset 1008,
> > > > flags [none], proto Options (0), length 478)
> > > >
> > > >     127.0.0.1 > 127.0.0.1: ip-proto-0
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Am I doing something wrong?! Or it is a bug.
> > > >
> > > > è As you see the tcpdump shows the offset of each fragment, and
> > > > testpmd prints L4_FRAG, so the both are recognizing that this is a
> > > fragmented packet.
> > >
> > > GRO library merges TSOed/GSOed packets, whose IP IDs and TCP
> > sequences
> > > are both consecutive. If input packets have same IP IDs, no packets
> > > will be merged.
> > >
> > > BTW, you can use iperf to test GRO feature.
> > >
> > > Best Regards,
> > > Jiayu
> > >
> > > >
> > > >
> > > >
> > > > Best regards,
> > > >
> > > > Wisam Jaddo
> > > >
> > > >
> > > >