From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) by dpdk.org (Postfix) with ESMTP id 502DD8E9D for ; Mon, 31 Aug 2015 08:29:58 +0200 (CEST) Received: by wicne3 with SMTP id ne3so64517822wic.0 for ; Sun, 30 Aug 2015 23:29:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:to:references:cc:from:message-id:date :user-agent:mime-version:in-reply-to:content-type :content-transfer-encoding; bh=UoYbTcjvN4ZgraKcqWWqlGpjSNymrsO8wSEfNEjBq8A=; b=aw60l9RqVQP4vq/LL1J9E9ExEgj5HIHMbUQCiP6qsJnfbkar59ra7Kx5CCIlfvHBZN djPPNOWwLJb0lVvV7hRfSmtnRrU6LF/p7dlHWkuVISWrY7rEhCxOSzaNSyXdzoq1y6iv 0vgAVvYPiXlQ1ZlHlKlNcla7ANIte/AVdfFgmaYavJP+DUF7HqGwXVl6iNCrJmQHDKqL vUuFcNnlir8kjKYq8LJ3IoMS1+UxrEttAdLC9A+QWHZDRMBgHv+j5+hUmeW76kNOdTnZ //ue5iY5S21ofc5lrhLsZVr2idMG5+s6zwFMF7v6y0Dudmq9LylaMj5CzWFBodL5mcnE +n8Q== X-Gm-Message-State: ALoCoQnj84ual1hvd3XeMJAUiUpyVsyvo8soYc3DtUdHSOgA1w3OVSWm/wxtaZomdFDiW6dYeo88 X-Received: by 10.194.119.161 with SMTP id kv1mr24244774wjb.157.1441002598165; Sun, 30 Aug 2015 23:29:58 -0700 (PDT) Received: from [10.0.0.4] (bzq-79-180-224-237.red.bezeqint.net. [79.180.224.237]) by smtp.googlemail.com with ESMTPSA id j2sm20582619wjq.5.2015.08.30.23.29.56 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sun, 30 Aug 2015 23:29:57 -0700 (PDT) To: Helin Zhang , "Ananyev, Konstantin" References: <55E2FF15.6020808@cloudius-systems.com> From: Vlad Zolotarov Message-ID: <55E3F464.4090408@cloudius-systems.com> Date: Mon, 31 Aug 2015 09:29:56 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <55E2FF15.6020808@cloudius-systems.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] i40e: XL710 Rx filters out frames above 1510 bytes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 31 Aug 2015 06:29:58 -0000 On 08/30/15 16:03, Vlad Zolotarov wrote: > Hi, I have the most strange issue on a setup of 2 pairs of connected > back to back XL710 cards. > It seems that frames above 1510 bytes are being filtered out by an > i40e PMD receiver. > The same setup works perfectly when I use Linux drivers on both sides > but when I use a PMD on one side and a Linux driver on the other - the > issue above occurs. > > i40e PMD statistics show nothing: all counters stay zero. > The MFS field in PRTGL_SAH register has the expected (reset) value: > 0x2600. We've found the problem. There is additional configuration per Rx queue context that limits the maximum allowed frame size - RXMAX. And it is configured to be 1518 by default. Which is supposed to be MTU + L2 HDR + CRC = 1500 + 14 + 4. However, when the MTU sized frames are being sent they are still being filtered out. This is probably due to some HW "feature" that requires RXMAX include VLAN header size even if there isn't any VLAN header. This "feature" has been addressed for instance in the Jesse's 61a46a4c0 commit in the net-next tree. In addition there is some mysterious "GLV_REPC/GLVREPC counter of the VSI" that is supposed to count this and a few other Rx drops events but this counter is nowhere to be found - neither in the data sheet (its offset on the bar I mean), nor in the DPDK i40e PMD registers description, nor in the net-next Linux driver. DPDK has a "/* GLV_REPC not supported */" in the i40e_ethdev.c implying that there must be some problem with this register too and it's a pity. I'll send a patch to the dpdk-dev that fixes the RXMAX issue shortly. thanks, vlad > > I even tried to disable all Tx offload capabilities on the Linux side > - still no success. > > Could u, guys clarify to be what is going on and what I may be doing > wrong? > Pls., let me know if u need any additional info about the setup. > > thanks in advance, > vlad > > > How to reproduce the issue: > > I used the in tree apps from examples/multi_process/client_server_mp: > > > On the DPDK box: > > 1. Bound both NICs to DPDK's UIO. > 2. Configured 1024 huge pages on each NUMA Node (there are two NUMA > Nodes in total). > > $ cat /proc/meminfo | grep Huge > AnonHugePages: 0 kB > HugePages_Total: 2048 > HugePages_Free: 292 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > > 3. Compiled mp_server and mp_client example applications. > 4. Run: sudo ./build/mp_server -c 6 -n 4 -- -p 0x3 -n 1 > 5. Run: sudo ./build/mp_client -c 8 -n 4 --proc-type=auto -- -n 0 > > > On the "other side" box (with the Linux device drivers): > > 1. configure a static arp: > > $ arp 192.169.10.118 > Address HWtype HWaddress Flags > Mask Iface > 192.169.10.118 ether 68:05:ca:2d:39:88 > CM ens3 > > 2. Set the MTU on ens3 to 3000 (see below). > 3. Use *ping* application for sending frames of different sizes: > 1. "ping 192.169.10.118 -s 1400" makes both Rx and Tx counters on > the DPDK side increment. > 2. "ping 192.169.10.118 -s 1500" - Rx counter on the DPDK side > doesn't increment. > > > The DPDK PMD driven setup description: > > * Two XL710 single port cards connected to XL710 cards in a different > server back to back. > > $ lspci | grep Ether > ---snip--- > 05:00.0 Ethernet controller: Intel Corporation Ethernet Controller > XL710 for 40GbE QSFP+ (rev 01) > 83:00.0 Ethernet controller: Intel Corporation Ethernet Controller > XL710 for 40GbE QSFP+ (rev 01) > ---snip--- > > $ ethtool -i ens288 > driver: i40e > version: 1.3.2-k > firmware-version: f4.22.26225 a1.1 n4.24 e13fd > bus-info: 0000:83:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > $ ethtool -i ens785 > driver: i40e > version: 1.3.2-k > firmware-version: f4.22.26225 a1.1 n4.24 e13fd > bus-info: 0000:05:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: yes > > $ ifconfig > ---snip--- > ens288: flags=4163 mtu 1500 > ether 68:05:ca:2d:3a:00 txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > ens785: flags=4163 mtu 1500 > ether 68:05:ca:2d:39:88 txqueuelen 1000 (Ethernet) > RX packets 0 bytes 0 (0.0 B) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 0 bytes 0 (0.0 B) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > ---snip--- > > > $ cat /etc/redhat-release > Fedora release 21 (Twenty One) > > * DPDK git HEAD 7173acefc7cfdfbbb9b91fcba1c9a67adb4c07c9 > > > The Linux drivers driven setup description: > > * Two XL710 single port cards connected to XL710 cards in a different > server back to back. > > $ lspci | grep Ether > ---snip--- > 04:00.0 Ethernet controller: Intel Corporation Ethernet Controller > XL710 for 40GbE QSFP+ (rev 01) > 83:00.0 Ethernet controller: Intel Corporation Ethernet Controller > XL710 for 40GbE QSFP+ (rev 01) > ---snip--- > > $ ethtool -i ens3 > driver: i40e > version: 1.2.6-k > firmware-version: f4.1 a1.1 n04.10 e80001121 > bus-info: 0000:83:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: no > > $ ethtool -i ens5 > driver: i40e > version: 1.2.6-k > firmware-version: f4.1 a1.1 n04.10 e80001121 > bus-info: 0000:04:00.0 > supports-statistics: yes > supports-test: yes > supports-eeprom-access: yes > supports-register-dump: yes > supports-priv-flags: no > > $ ifconfig > ---snip--- > ens3: flags=4163 mtu 3000 > inet 192.169.10.199 netmask 255.255.255.0 broadcast > 192.169.10.255 > ether 68:05:ca:27:d3:72 txqueuelen 1000 (Ethernet) > RX packets 69 bytes 30155 (29.4 KiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 746 bytes 820489 (801.2 KiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > ens5: flags=4163 mtu 1500 > inet 1.1.1.2 netmask 255.255.255.0 broadcast 1.1.1.255 > inet6 fe80::6a05:caff:fe27:d3ea prefixlen 64 scopeid 0x20 > ether 68:05:ca:27:d3:ea txqueuelen 1000 (Ethernet) > RX packets 523572634 bytes 34869102046 (32.4 GiB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 523254659 bytes 153342981238 (142.8 GiB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > ---snip--- > > $ cat /etc/redhat-release > Fedora release 21 (Twenty One)_ > > _$ modinfo i40e > filename: > /lib/modules/4.0.6-200.fc21.x86_64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko.xz > version: 1.2.6-k > license: GPL > description: Intel(R) Ethernet Connection XL710 Network Driver > author: Intel Corporation, > srcversion: 14800421FE3354248473729 > alias: pci:v00008086d00001586sv*sd*bc*sc*i* > alias: pci:v00008086d00001585sv*sd*bc*sc*i* > alias: pci:v00008086d00001584sv*sd*bc*sc*i* > alias: pci:v00008086d00001583sv*sd*bc*sc*i* > alias: pci:v00008086d00001581sv*sd*bc*sc*i* > alias: pci:v00008086d00001580sv*sd*bc*sc*i* > alias: pci:v00008086d0000157Fsv*sd*bc*sc*i* > alias: pci:v00008086d00001574sv*sd*bc*sc*i* > alias: pci:v00008086d00001572sv*sd*bc*sc*i* > depends: ptp,vxlan > intree: Y > vermagic: 4.0.6-200.fc21.x86_64 SMP mod_unload > signer: Fedora kernel signing key > sig_key: F9:FE:41:30:BC:4F:80:F6:C4:1E:A1:E1:5C:73:37:9B:B8:BA:C8:4C > sig_hashalgo: sha256 > parm: debug:Debug level (0=none,...,16=all) (int) > _ > _