Re: [dpdk-dev] i40e: XL710 Rx filters out frames above 1510 bytes

[Vlad Zolotarov <vladz@cloudius-systems.com>]

On 08/30/15 16:03, Vlad Zolotarov wrote:
> Hi, I have the most strange issue on a setup of 2 pairs of connected 
> back to back XL710 cards.
> It seems that frames above 1510 bytes are being filtered out by an 
> i40e PMD receiver.
> The same setup works perfectly when I use Linux drivers on both sides 
> but when I use a PMD on one side and a Linux driver on the other - the 
> issue above occurs.
>
> i40e PMD statistics show nothing: all counters stay zero.
> The MFS field in PRTGL_SAH register has the expected (reset) value: 
> 0x2600.

We've found the problem. There is additional configuration per Rx queue 
context that limits the maximum
allowed frame size - RXMAX. And it is configured to be 1518 by default. 
Which is supposed to be MTU + L2 HDR + CRC = 1500 + 14 + 4. However,
when the MTU sized frames are being sent they are still being filtered 
out. This is probably due to some HW "feature" that requires RXMAX 
include VLAN header size even if there isn't any VLAN header. This 
"feature" has been addressed for instance in the Jesse's 61a46a4c0 
commit in the net-next tree.

In addition there is some mysterious "GLV_REPC/GLVREPC counter of the 
VSI" that is supposed to count this and a few other Rx drops events but 
this counter is nowhere to be found - neither in the data sheet (its 
offset on the bar I mean), nor in the DPDK i40e PMD registers 
description, nor in the net-next Linux driver. DPDK has a "/* GLV_REPC 
not supported */" in the i40e_ethdev.c implying that there must be some 
problem with this register too and it's a pity.

I'll send a patch to the dpdk-dev that fixes the RXMAX issue shortly.

thanks,
vlad

>
> I even tried to disable all Tx offload capabilities on the Linux side 
> - still no success.
>
> Could u, guys clarify to be what is going on and what I may be doing 
> wrong?
> Pls., let me know if u need any additional info about the setup.
>
> thanks in advance,
> vlad
>
>
>        How to reproduce the issue:
>
> I used the in tree apps from examples/multi_process/client_server_mp:
>
>
>          On the DPDK box:
>
> 1. Bound both NICs to DPDK's UIO.
> 2. Configured 1024 huge pages on each NUMA Node (there are two NUMA
>    Nodes in total).
>
> $ cat /proc/meminfo  | grep Huge
> AnonHugePages:         0 kB
> HugePages_Total:    2048
> HugePages_Free:      292
> HugePages_Rsvd:        0
> HugePages_Surp:        0
> Hugepagesize:       2048 kB
>
> 3. Compiled mp_server and mp_client example applications.
> 4. Run: sudo ./build/mp_server -c 6 -n 4  --  -p 0x3 -n 1
> 5. Run: sudo ./build/mp_client -c 8 -n 4 --proc-type=auto -- -n 0
>
>
>          On the "other side" box (with the Linux device drivers):
>
> 1. configure a static arp:
>
> $ arp 192.169.10.118
> Address                  HWtype  HWaddress           Flags 
> Mask            Iface
> 192.169.10.118           ether   68:05:ca:2d:39:88 
> CM                    ens3
>
> 2. Set the MTU on ens3 to 3000 (see below).
> 3. Use *ping* application for sending frames of different sizes:
>     1. "ping 192.169.10.118 -s 1400" makes both Rx and Tx counters on
>        the DPDK side increment.
>     2. "ping 192.169.10.118 -s 1500" - Rx counter on the DPDK side
>        doesn't increment.
>
>
>        The DPDK PMD driven setup description:
>
>  * Two XL710 single port cards connected to XL710 cards in a different
>    server back to back.
>
> $ lspci | grep Ether
> ---snip---
> 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
> XL710 for 40GbE QSFP+ (rev 01)
> 83:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
> XL710 for 40GbE QSFP+ (rev 01)
> ---snip---
>
> $ ethtool -i ens288
> driver: i40e
> version: 1.3.2-k
> firmware-version: f4.22.26225 a1.1 n4.24 e13fd
> bus-info: 0000:83:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> $ ethtool -i ens785
> driver: i40e
> version: 1.3.2-k
> firmware-version: f4.22.26225 a1.1 n4.24 e13fd
> bus-info: 0000:05:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> $ ifconfig
> ---snip---
> ens288: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         ether 68:05:ca:2d:3a:00  txqueuelen 1000  (Ethernet)
>         RX packets 0  bytes 0 (0.0 B)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 0  bytes 0 (0.0 B)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>
> ens785: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         ether 68:05:ca:2d:39:88  txqueuelen 1000  (Ethernet)
>         RX packets 0  bytes 0 (0.0 B)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 0  bytes 0 (0.0 B)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
> ---snip---
>
>
> $ cat /etc/redhat-release
> Fedora release 21 (Twenty One)
>
>  * DPDK git HEAD 7173acefc7cfdfbbb9b91fcba1c9a67adb4c07c9
>
>
>        The Linux drivers driven setup description:
>
>  * Two XL710 single port cards connected to XL710 cards in a different
>    server back to back.
>
> $ lspci | grep Ether
> ---snip---
> 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
> XL710 for 40GbE QSFP+ (rev 01)
> 83:00.0 Ethernet controller: Intel Corporation Ethernet Controller 
> XL710 for 40GbE QSFP+ (rev 01)
> ---snip---
>
> $ ethtool -i ens3
> driver: i40e
> version: 1.2.6-k
> firmware-version: f4.1 a1.1 n04.10 e80001121
> bus-info: 0000:83:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> $ ethtool -i ens5
> driver: i40e
> version: 1.2.6-k
> firmware-version: f4.1 a1.1 n04.10 e80001121
> bus-info: 0000:04:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> $ ifconfig
> ---snip---
> ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 3000
>         inet 192.169.10.199  netmask 255.255.255.0  broadcast 
> 192.169.10.255
>         ether 68:05:ca:27:d3:72  txqueuelen 1000  (Ethernet)
>         RX packets 69  bytes 30155 (29.4 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 746  bytes 820489 (801.2 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>
> ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet 1.1.1.2  netmask 255.255.255.0  broadcast 1.1.1.255
>         inet6 fe80::6a05:caff:fe27:d3ea  prefixlen 64  scopeid 0x20<link>
>         ether 68:05:ca:27:d3:ea  txqueuelen 1000  (Ethernet)
>         RX packets 523572634  bytes 34869102046 (32.4 GiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 523254659  bytes 153342981238 (142.8 GiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>
> ---snip---
>
> $ cat /etc/redhat-release
> Fedora release 21 (Twenty One)_
>
> _$ modinfo i40e
> filename: 
> /lib/modules/4.0.6-200.fc21.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
> version:        1.2.6-k
> license:        GPL
> description:    Intel(R) Ethernet Connection XL710 Network Driver
> author:         Intel Corporation, <e1000-devel@lists.sourceforge.net>
> srcversion:     14800421FE3354248473729
> alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
> alias:          pci:v00008086d0000157Fsv*sd*bc*sc*i*
> alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
> alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
> depends:        ptp,vxlan
> intree:         Y
> vermagic:       4.0.6-200.fc21.x86_64 SMP mod_unload
> signer:         Fedora kernel signing key
> sig_key: F9:FE:41:30:BC:4F:80:F6:C4:1E:A1:E1:5C:73:37:9B:B8:BA:C8:4C
> sig_hashalgo:   sha256
> parm:           debug:Debug level (0=none,...,16=all) (int)
> _
> _