* [dpdk-dev] i40e: XL710 Rx filters out frames above 1510 bytes
@ 2015-08-30 13:03 Vlad Zolotarov
2015-08-31 6:29 ` Vlad Zolotarov
0 siblings, 1 reply; 2+ messages in thread
From: Vlad Zolotarov @ 2015-08-30 13:03 UTC (permalink / raw)
To: Helin Zhang, Ananyev, Konstantin; +Cc: dev
Hi, I have the most strange issue on a setup of 2 pairs of connected
back to back XL710 cards.
It seems that frames above 1510 bytes are being filtered out by an i40e
PMD receiver.
The same setup works perfectly when I use Linux drivers on both sides
but when I use a PMD on one side and a Linux driver on the other - the
issue above occurs.
i40e PMD statistics show nothing: all counters stay zero.
The MFS field in PRTGL_SAH register has the expected (reset) value: 0x2600.
I even tried to disable all Tx offload capabilities on the Linux side -
still no success.
Could u, guys clarify to be what is going on and what I may be doing wrong?
Pls., let me know if u need any additional info about the setup.
thanks in advance,
vlad
How to reproduce the issue:
I used the in tree apps from examples/multi_process/client_server_mp:
On the DPDK box:
1. Bound both NICs to DPDK's UIO.
2. Configured 1024 huge pages on each NUMA Node (there are two NUMA
Nodes in total).
$ cat /proc/meminfo | grep Huge
AnonHugePages: 0 kB
HugePages_Total: 2048
HugePages_Free: 292
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
3. Compiled mp_server and mp_client example applications.
4. Run: sudo ./build/mp_server -c 6 -n 4 -- -p 0x3 -n 1
5. Run: sudo ./build/mp_client -c 8 -n 4 --proc-type=auto -- -n 0
On the "other side" box (with the Linux device drivers):
1. configure a static arp:
$ arp 192.169.10.118
Address HWtype HWaddress Flags
Mask Iface
192.169.10.118 ether 68:05:ca:2d:39:88 CM
ens3
2. Set the MTU on ens3 to 3000 (see below).
3. Use *ping* application for sending frames of different sizes:
1. "ping 192.169.10.118 -s 1400" makes both Rx and Tx counters on
the DPDK side increment.
2. "ping 192.169.10.118 -s 1500" - Rx counter on the DPDK side
doesn't increment.
The DPDK PMD driven setup description:
* Two XL710 single port cards connected to XL710 cards in a different
server back to back.
$ lspci | grep Ether
---snip---
05:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710
for 40GbE QSFP+ (rev 01)
83:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710
for 40GbE QSFP+ (rev 01)
---snip---
$ ethtool -i ens288
driver: i40e
version: 1.3.2-k
firmware-version: f4.22.26225 a1.1 n4.24 e13fd
bus-info: 0000:83:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
$ ethtool -i ens785
driver: i40e
version: 1.3.2-k
firmware-version: f4.22.26225 a1.1 n4.24 e13fd
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
$ ifconfig
---snip---
ens288: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 68:05:ca:2d:3a:00 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens785: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
ether 68:05:ca:2d:39:88 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
---snip---
$ cat /etc/redhat-release
Fedora release 21 (Twenty One)
* DPDK git HEAD 7173acefc7cfdfbbb9b91fcba1c9a67adb4c07c9
The Linux drivers driven setup description:
* Two XL710 single port cards connected to XL710 cards in a different
server back to back.
$ lspci | grep Ether
---snip---
04:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710
for 40GbE QSFP+ (rev 01)
83:00.0 Ethernet controller: Intel Corporation Ethernet Controller XL710
for 40GbE QSFP+ (rev 01)
---snip---
$ ethtool -i ens3
driver: i40e
version: 1.2.6-k
firmware-version: f4.1 a1.1 n04.10 e80001121
bus-info: 0000:83:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
$ ethtool -i ens5
driver: i40e
version: 1.2.6-k
firmware-version: f4.1 a1.1 n04.10 e80001121
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no
$ ifconfig
---snip---
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 3000
inet 192.169.10.199 netmask 255.255.255.0 broadcast
192.169.10.255
ether 68:05:ca:27:d3:72 txqueuelen 1000 (Ethernet)
RX packets 69 bytes 30155 (29.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 746 bytes 820489 (801.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 1.1.1.2 netmask 255.255.255.0 broadcast 1.1.1.255
inet6 fe80::6a05:caff:fe27:d3ea prefixlen 64 scopeid 0x20<link>
ether 68:05:ca:27:d3:ea txqueuelen 1000 (Ethernet)
RX packets 523572634 bytes 34869102046 (32.4 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 523254659 bytes 153342981238 (142.8 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
---snip---
$ cat /etc/redhat-release
Fedora release 21 (Twenty One)_
_$ modinfo i40e
filename:
/lib/modules/4.0.6-200.fc21.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
version: 1.2.6-k
license: GPL
description: Intel(R) Ethernet Connection XL710 Network Driver
author: Intel Corporation, <e1000-devel@lists.sourceforge.net>
srcversion: 14800421FE3354248473729
alias: pci:v00008086d00001586sv*sd*bc*sc*i*
alias: pci:v00008086d00001585sv*sd*bc*sc*i*
alias: pci:v00008086d00001584sv*sd*bc*sc*i*
alias: pci:v00008086d00001583sv*sd*bc*sc*i*
alias: pci:v00008086d00001581sv*sd*bc*sc*i*
alias: pci:v00008086d00001580sv*sd*bc*sc*i*
alias: pci:v00008086d0000157Fsv*sd*bc*sc*i*
alias: pci:v00008086d00001574sv*sd*bc*sc*i*
alias: pci:v00008086d00001572sv*sd*bc*sc*i*
depends: ptp,vxlan
intree: Y
vermagic: 4.0.6-200.fc21.x86_64 SMP mod_unload
signer: Fedora kernel signing key
sig_key: F9:FE:41:30:BC:4F:80:F6:C4:1E:A1:E1:5C:73:37:9B:B8:BA:C8:4C
sig_hashalgo: sha256
parm: debug:Debug level (0=none,...,16=all) (int)
_
_
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: [dpdk-dev] i40e: XL710 Rx filters out frames above 1510 bytes
2015-08-30 13:03 [dpdk-dev] i40e: XL710 Rx filters out frames above 1510 bytes Vlad Zolotarov
@ 2015-08-31 6:29 ` Vlad Zolotarov
0 siblings, 0 replies; 2+ messages in thread
From: Vlad Zolotarov @ 2015-08-31 6:29 UTC (permalink / raw)
To: Helin Zhang, Ananyev, Konstantin; +Cc: dev
On 08/30/15 16:03, Vlad Zolotarov wrote:
> Hi, I have the most strange issue on a setup of 2 pairs of connected
> back to back XL710 cards.
> It seems that frames above 1510 bytes are being filtered out by an
> i40e PMD receiver.
> The same setup works perfectly when I use Linux drivers on both sides
> but when I use a PMD on one side and a Linux driver on the other - the
> issue above occurs.
>
> i40e PMD statistics show nothing: all counters stay zero.
> The MFS field in PRTGL_SAH register has the expected (reset) value:
> 0x2600.
We've found the problem. There is additional configuration per Rx queue
context that limits the maximum
allowed frame size - RXMAX. And it is configured to be 1518 by default.
Which is supposed to be MTU + L2 HDR + CRC = 1500 + 14 + 4. However,
when the MTU sized frames are being sent they are still being filtered
out. This is probably due to some HW "feature" that requires RXMAX
include VLAN header size even if there isn't any VLAN header. This
"feature" has been addressed for instance in the Jesse's 61a46a4c0
commit in the net-next tree.
In addition there is some mysterious "GLV_REPC/GLVREPC counter of the
VSI" that is supposed to count this and a few other Rx drops events but
this counter is nowhere to be found - neither in the data sheet (its
offset on the bar I mean), nor in the DPDK i40e PMD registers
description, nor in the net-next Linux driver. DPDK has a "/* GLV_REPC
not supported */" in the i40e_ethdev.c implying that there must be some
problem with this register too and it's a pity.
I'll send a patch to the dpdk-dev that fixes the RXMAX issue shortly.
thanks,
vlad
>
> I even tried to disable all Tx offload capabilities on the Linux side
> - still no success.
>
> Could u, guys clarify to be what is going on and what I may be doing
> wrong?
> Pls., let me know if u need any additional info about the setup.
>
> thanks in advance,
> vlad
>
>
> How to reproduce the issue:
>
> I used the in tree apps from examples/multi_process/client_server_mp:
>
>
> On the DPDK box:
>
> 1. Bound both NICs to DPDK's UIO.
> 2. Configured 1024 huge pages on each NUMA Node (there are two NUMA
> Nodes in total).
>
> $ cat /proc/meminfo | grep Huge
> AnonHugePages: 0 kB
> HugePages_Total: 2048
> HugePages_Free: 292
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
>
> 3. Compiled mp_server and mp_client example applications.
> 4. Run: sudo ./build/mp_server -c 6 -n 4 -- -p 0x3 -n 1
> 5. Run: sudo ./build/mp_client -c 8 -n 4 --proc-type=auto -- -n 0
>
>
> On the "other side" box (with the Linux device drivers):
>
> 1. configure a static arp:
>
> $ arp 192.169.10.118
> Address HWtype HWaddress Flags
> Mask Iface
> 192.169.10.118 ether 68:05:ca:2d:39:88
> CM ens3
>
> 2. Set the MTU on ens3 to 3000 (see below).
> 3. Use *ping* application for sending frames of different sizes:
> 1. "ping 192.169.10.118 -s 1400" makes both Rx and Tx counters on
> the DPDK side increment.
> 2. "ping 192.169.10.118 -s 1500" - Rx counter on the DPDK side
> doesn't increment.
>
>
> The DPDK PMD driven setup description:
>
> * Two XL710 single port cards connected to XL710 cards in a different
> server back to back.
>
> $ lspci | grep Ether
> ---snip---
> 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> XL710 for 40GbE QSFP+ (rev 01)
> 83:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> XL710 for 40GbE QSFP+ (rev 01)
> ---snip---
>
> $ ethtool -i ens288
> driver: i40e
> version: 1.3.2-k
> firmware-version: f4.22.26225 a1.1 n4.24 e13fd
> bus-info: 0000:83:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> $ ethtool -i ens785
> driver: i40e
> version: 1.3.2-k
> firmware-version: f4.22.26225 a1.1 n4.24 e13fd
> bus-info: 0000:05:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: yes
>
> $ ifconfig
> ---snip---
> ens288: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> ether 68:05:ca:2d:3a:00 txqueuelen 1000 (Ethernet)
> RX packets 0 bytes 0 (0.0 B)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 0 bytes 0 (0.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> ens785: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> ether 68:05:ca:2d:39:88 txqueuelen 1000 (Ethernet)
> RX packets 0 bytes 0 (0.0 B)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 0 bytes 0 (0.0 B)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
> ---snip---
>
>
> $ cat /etc/redhat-release
> Fedora release 21 (Twenty One)
>
> * DPDK git HEAD 7173acefc7cfdfbbb9b91fcba1c9a67adb4c07c9
>
>
> The Linux drivers driven setup description:
>
> * Two XL710 single port cards connected to XL710 cards in a different
> server back to back.
>
> $ lspci | grep Ether
> ---snip---
> 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> XL710 for 40GbE QSFP+ (rev 01)
> 83:00.0 Ethernet controller: Intel Corporation Ethernet Controller
> XL710 for 40GbE QSFP+ (rev 01)
> ---snip---
>
> $ ethtool -i ens3
> driver: i40e
> version: 1.2.6-k
> firmware-version: f4.1 a1.1 n04.10 e80001121
> bus-info: 0000:83:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> $ ethtool -i ens5
> driver: i40e
> version: 1.2.6-k
> firmware-version: f4.1 a1.1 n04.10 e80001121
> bus-info: 0000:04:00.0
> supports-statistics: yes
> supports-test: yes
> supports-eeprom-access: yes
> supports-register-dump: yes
> supports-priv-flags: no
>
> $ ifconfig
> ---snip---
> ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 3000
> inet 192.169.10.199 netmask 255.255.255.0 broadcast
> 192.169.10.255
> ether 68:05:ca:27:d3:72 txqueuelen 1000 (Ethernet)
> RX packets 69 bytes 30155 (29.4 KiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 746 bytes 820489 (801.2 KiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
> inet 1.1.1.2 netmask 255.255.255.0 broadcast 1.1.1.255
> inet6 fe80::6a05:caff:fe27:d3ea prefixlen 64 scopeid 0x20<link>
> ether 68:05:ca:27:d3:ea txqueuelen 1000 (Ethernet)
> RX packets 523572634 bytes 34869102046 (32.4 GiB)
> RX errors 0 dropped 0 overruns 0 frame 0
> TX packets 523254659 bytes 153342981238 (142.8 GiB)
> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>
> ---snip---
>
> $ cat /etc/redhat-release
> Fedora release 21 (Twenty One)_
>
> _$ modinfo i40e
> filename:
> /lib/modules/4.0.6-200.fc21.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz
> version: 1.2.6-k
> license: GPL
> description: Intel(R) Ethernet Connection XL710 Network Driver
> author: Intel Corporation, <e1000-devel@lists.sourceforge.net>
> srcversion: 14800421FE3354248473729
> alias: pci:v00008086d00001586sv*sd*bc*sc*i*
> alias: pci:v00008086d00001585sv*sd*bc*sc*i*
> alias: pci:v00008086d00001584sv*sd*bc*sc*i*
> alias: pci:v00008086d00001583sv*sd*bc*sc*i*
> alias: pci:v00008086d00001581sv*sd*bc*sc*i*
> alias: pci:v00008086d00001580sv*sd*bc*sc*i*
> alias: pci:v00008086d0000157Fsv*sd*bc*sc*i*
> alias: pci:v00008086d00001574sv*sd*bc*sc*i*
> alias: pci:v00008086d00001572sv*sd*bc*sc*i*
> depends: ptp,vxlan
> intree: Y
> vermagic: 4.0.6-200.fc21.x86_64 SMP mod_unload
> signer: Fedora kernel signing key
> sig_key: F9:FE:41:30:BC:4F:80:F6:C4:1E:A1:E1:5C:73:37:9B:B8:BA:C8:4C
> sig_hashalgo: sha256
> parm: debug:Debug level (0=none,...,16=all) (int)
> _
> _
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2015-08-31 6:29 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-08-30 13:03 [dpdk-dev] i40e: XL710 Rx filters out frames above 1510 bytes Vlad Zolotarov
2015-08-31 6:29 ` Vlad Zolotarov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).