From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-f43.google.com (mail-it0-f43.google.com [209.85.214.43]) by dpdk.org (Postfix) with ESMTP id 7B84A4AAD for ; Fri, 22 Jul 2016 14:35:59 +0200 (CEST) Received: by mail-it0-f43.google.com with SMTP id j124so45574389ith.1 for ; Fri, 22 Jul 2016 05:35:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=hLV+20yrLeyonvFosMtKVXsAbui6dwwiLnZ06j3vpuU=; b=hlUl36MAqGF752cZftTUZrmaLc10+ATFH56TSYiI9mwZ4YHpR6vGNgN5SUnHwhZpwC qLzrJrl+Flu9uPLqgssT5wZ4T34VnLYhynzyu54WgBNzJx7R84wGulHh6LghUt8it91S FiZ3ziaMmcS3TKqO8YVhk83qKRNGqEgc4zmWCeGGDWmADCS1XNJqetsfB1v94kkfaB+7 o9j6D5dnHhl8uWFeVy/zA7SUE4pXjTGDYrBpzTp73vepvQ1QrYyvZDBIivBsKROIbl91 MkCark7t7PFlBV5H8l8kLlF+8VCtzuNTnnEEoA788FaWQbfXoZEYgANDnRcUkeHggWmX oPiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=hLV+20yrLeyonvFosMtKVXsAbui6dwwiLnZ06j3vpuU=; b=JfvxpPqRsJtxe612Xr4z33oMTmGVMeZREmAcph3ImbRheYesaGEpXQ9b6hI4yfHoO1 Tajlx2iHBNopO5h3QsSFRXJHJvJwSPM9g4LvTb4i0lWoqGKLhOj65JrnLw9OnjzczXfx dFsaqvMsHTyjM29muKbZATBH+rI7nrUjHC9/vUllYQxwe4muf1nhPx/frDA2UoT7cVN3 XrPzflFYbuO4YrMSYcFPpioD2EcaG8aYG96NTCYHROs6e5ltnRlmLhl6ppJ70kt8dqOF LXpou8Ygo/kU+fV4RQKLjTMgzlZigxUwuhWaniYbs7rCiUqHyO2Rb2dx6RXFpwugoU3g KYXw== X-Gm-Message-State: ALyK8tJ77t3JwoRaxpbAOII5vhUjZtMn6XjOIyNznmILR4xLfgY207jktcL8p7AYuiHD2GHy62ihB6Sfl/jhNw== X-Received: by 10.36.242.68 with SMTP id j65mr19886564ith.25.1469190958884; Fri, 22 Jul 2016 05:35:58 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.7.82 with HTTP; Fri, 22 Jul 2016 05:35:39 -0700 (PDT) In-Reply-To: References: <94479800C636CB44BD422CB454846E013B5A15@SHSMSX101.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013B5ED3@SHSMSX101.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013B7479@SHSMSX101.ccr.corp.intel.com> From: Take Ceara Date: Fri, 22 Jul 2016 14:35:39 +0200 Message-ID: To: "Xing, Beilei" Cc: "Zhang, Helin" , "Wu, Jingjing" , "dev@dpdk.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 12:35:59 -0000 On Fri, Jul 22, 2016 at 2:31 PM, Take Ceara wrote= : > Hi Beilei, > > On Fri, Jul 22, 2016 at 11:04 AM, Xing, Beilei wr= ote: >> Hi Ceara, >> >>> -----Original Message----- >>> From: Take Ceara [mailto:dumitru.ceara@gmail.com] >>> Sent: Thursday, July 21, 2016 6:58 PM >>> To: Xing, Beilei >>> Cc: Zhang, Helin ; Wu, Jingjing >>> ; dev@dpdk.org >>> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X71= 0 >>> NICs for some RX mbuf sizes >>> >>> >>> Following your testpmd example run I managed to replicate the problem o= n >>> my dpdk 16.04 setup like this: >>> >>> I have two X710 adapters connected back to back: >>> $ ./tools/dpdk_nic_bind.py -s >>> >>> Network devices using DPDK-compatible driver >>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>> 0000:01:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=3Digb_uio un= used=3D >>> 0000:81:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=3Digb_uio un= used=3D >>> >>> The firmware of the two adapters is up to date with the latest >>> version: 5.04 (f5.0.40043 a1.5 n5.04 e24cd) >>> >>> I run testpmd with mbuf-size 1152 and txpktsize 1024 such that upon rec= eival >>> the whole mbuf (except headroom) is filled. >>> I enabled RX IP checksum in hw and RX RSS hashing for UDP. >>> With test-pmd forward mode "rxonly" and verbose 1 I see that incoming >>> packets have PKT_RX_RSS_HASH set but the hash value is 0. >>> >>> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >>> --coremask=3D0xffff0 --rxq=3D16 --txq=3D16 --mbuf-size 1152 --txpkts 10= 24 -- >>> enable-rx-cksum --rss-udp [...] >>> testpmd> set verbose 1 >>> Change verbose level from 0 to 1 >>> testpmd> set fwd rxonly >>> Set rxonly packet forwarding mode >>> testpmd> start tx_first >>> rxonly packet forwarding - CRC stripping disabled - packets/burst=3D3= 2 >>> nb forwarding cores=3D16 - nb forwarding ports=3D2 >>> RX queues=3D16 - RX desc=3D128 - RX free threshold=3D32 >>> RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 >>> TX queues=3D16 - TX desc=3D512 - TX free threshold=3D32 >>> TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 >>> TX RS bit threshold=3D32 - TXQ flags=3D0xf01 port 0/queue 1: received= 32 >>> packets >>> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >>> length=3D1024 - nb_segs=3D1 - RSS hash=3D0x0 - RSS queue=3D0x1 - (outer= ) L2 >>> type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP >>> - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: >>> Unknown - Inner L4 type: Unknown >>> - Receive queue=3D0x1 >>> PKT_RX_RSS_HASH >>> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >>> length=3D1024 - nb_segs=3D1 - RSS hash=3D0x0 - RSS queue=3D0x1 - (outer= ) L2 >>> type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP >>> - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: >>> Unknown - Inner L4 type: Unknown >>> - Receive queue=3D0x1 >>> PKT_RX_RSS_HASH >> >> What's the source ip address and destination ip address of the packet yo= u sent to port 0? Could you try to change ip address or port number to obse= rve if hash value changes? I remember I saw hash value was 0 before, but wi= th different ip address, there'll be different hash values. > > I was using the test-pmd "txonly" implementation which sends fixed UDP > packets from 192.168.0.1:1024 -> 192.168.0.2:1024. > > I changed the test-pmd tx_only code so that it sends traffic with > incremental destination IP: 192.168.0.1:1024 -> [192.168.0.2, > 192.168.0.12]:1024 > I also dumped the source and destination IPs in the "rxonly" > pkt_burst_receive function. > Then I see that packets are indeed sent to different queues but the > mbuf->hash.rss value is still 0. > > ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i > --coremask=3D0xffff0 --rxq=3D16 --txq=3D16 --mbuf-size 1152 --txpkts 1024 > --enable-rx-cksum --rss-udp > > [...] > > - Receive queue=3D0xf > PKT_RX_RSS_HASH > src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - > length=3D1024 - nb_segs=3D1 - RSS queue=3D0xa - (outer) L2 type: ETHER - > (outer) L3 type: IPV4_EXT_UNKNOWN SIP=3DC0A80001 DIP=3DC0A80006 - (outer) > L4 type: UDP - Tunnel type: Unknown - RSS hash=3D0x0 - Inner L2 type: > Unknown - RSS queue=3D0xf - RSS queue=3D0x7 - (outer) L2 type: ETHER - > (outer) L3 type: IPV4_EXT_UNKNOWN SIP=3DC0A80001 DIP=3DC0A80007 - (outer) > L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner > L3 type: Unknown - Inner L4 type: Unknown > - Receive queue=3D0x7 > PKT_RX_RSS_HASH > src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - (outer) L2 type: > ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=3DC0A80001 DIP=3DC0A80009 - > type=3D0x0800 - length=3D1024 - nb_segs=3D1 - Inner L3 type: Unknown - In= ner > L4 type: Unknown - RSS hash=3D0x0 - (outer) L4 type: UDP - Tunnel type: > Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - RSS > queue=3D0x7 - Inner L4 type: Unknown > > [...] > > testpmd> stop > ------- Forward Stats for RX Port=3D 0/Queue=3D 0 -> TX Port=3D 1/Queue= =3D 0 ------- > RX-packets: 0 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 1 -> TX Port=3D 1/Queue= =3D 1 ------- > RX-packets: 59 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 2 -> TX Port=3D 1/Queue= =3D 2 ------- > RX-packets: 48 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 3 -> TX Port=3D 1/Queue= =3D 3 ------- > RX-packets: 0 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 4 -> TX Port=3D 1/Queue= =3D 4 ------- > RX-packets: 59 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 5 -> TX Port=3D 1/Queue= =3D 5 ------- > RX-packets: 0 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 6 -> TX Port=3D 1/Queue= =3D 6 ------- > RX-packets: 0 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 7 -> TX Port=3D 1/Queue= =3D 7 ------- > RX-packets: 48 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 8 -> TX Port=3D 1/Queue= =3D 8 ------- > RX-packets: 0 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D 9 -> TX Port=3D 1/Queue= =3D 9 ------- > RX-packets: 59 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D10 -> TX Port=3D 1/Queue= =3D10 ------- > RX-packets: 48 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D11 -> TX Port=3D 1/Queue= =3D11 ------- > RX-packets: 0 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D12 -> TX Port=3D 1/Queue= =3D12 ------- > RX-packets: 59 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D13 -> TX Port=3D 1/Queue= =3D13 ------- > RX-packets: 0 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D14 -> TX Port=3D 1/Queue= =3D14 ------- > RX-packets: 0 TX-packets: 32 TX-dropped: 0 > ------- Forward Stats for RX Port=3D 0/Queue=3D15 -> TX Port=3D 1/Queue= =3D15 ------- > RX-packets: 48 TX-packets: 32 TX-dropped: 0 > ---------------------- Forward statistics for port 0 -----------------= ----- > RX-packets: 428 RX-dropped: 84 RX-total: 512 > TX-packets: 0 TX-dropped: 0 TX-total: 0 > -----------------------------------------------------------------------= ----- > > ---------------------- Forward statistics for port 1 -----------------= ----- > RX-packets: 0 RX-dropped: 0 RX-total: 0 > TX-packets: 512 TX-dropped: 0 TX-total: 512 > -----------------------------------------------------------------------= ----- > > +++++++++++++++ Accumulated forward statistics for all ports+++++++++++= ++++ > RX-packets: 428 RX-dropped: 84 RX-total: 512 > TX-packets: 512 TX-dropped: 0 TX-total: 512 > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++++ > > I checked all the RSS hash values for all the 10 different incoming > streams and they're all 0. Also, the fact that the traffic is actually > distributed seems to suggest that RSS itself is working but the mbuf > hash field is (I guess) either not written or corrupted. > >> >>> >>> If I use a different mbuf-size, for example 2048, everything is fine: >>> >>> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >>> --coremask=3D0xffff0 --rxq=3D16 --txq=3D16 --mbuf-size 2048 --txpkts 10= 24 -- >>> enable-rx-cksum --rss-udp [...] >>> testpmd> set verbose 1 >>> Change verbose level from 0 to 1 >>> testpmd> set fwd rxonly >>> Set rxonly packet forwarding mode >>> testpmd> start tx_first >>> rxonly packet forwarding - CRC stripping disabled - packets/burst=3D3= 2 >>> nb forwarding cores=3D16 - nb forwarding ports=3D2 >>> RX queues=3D16 - RX desc=3D128 - RX free threshold=3D32 >>> RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 >>> TX queues=3D16 - TX desc=3D512 - TX free threshold=3D32 >>> TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 >>> TX RS bit threshold=3D32 - TXQ flags=3D0xf01 port 0/queue 1: received= 32 >>> packets >>> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >>> length=3D1024 - nb_segs=3D1 - RSS hash=3D0x5263c3f2 - RSS queue=3D0x1 - >>> (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) >>> L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner >>> L3 type: Unknown - Inner L4 type: Unknown >>> - Receive queue=3D0x1 >>> PKT_RX_RSS_HASH >>> >> >> Did you send the same packet as before to port 0? >> Sorry, I forgot to reply to this question. Yes, the packet is identical. >>> Another weird thing I noticed is when I run test-pmd without --enable-r= x- >>> cksum (which is the default mode) then the RSS flag doesn get set at al= l. >>> Instead the PKT_RX_FDIR flag gets set. This happens even though fdir_mo= de >>> is set to RTE_FDIR_MODE_NONE in the device >>> configuration: >>> >>> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >>> --coremask=3D0xffff0 --rxq=3D16 --txq=3D16 --mbuf-size 1152 --txpkts 10= 24 --rss- >>> udp [...] >>> testpmd> set verbose 1 >>> Change verbose level from 0 to 1 >>> testpmd> set fwd rxonly >>> Set rxonly packet forwarding mode >>> testpmd> start tx_first >>> rxonly packet forwarding - CRC stripping disabled - packets/burst=3D3= 2 >>> nb forwarding cores=3D16 - nb forwarding ports=3D2 >>> RX queues=3D16 - RX desc=3D128 - RX free threshold=3D32 >>> RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 >>> TX queues=3D16 - TX desc=3D512 - TX free threshold=3D32 >>> TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 >>> TX RS bit threshold=3D32 - TXQ flags=3D0xf01 port 0/queue 1: received= 16 >>> packets >>> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >>> length=3D1024 - nb_segs=3D2 - FDIR matched hash=3D0xc3f2 ID=3D0x5263 Un= known >>> packet type >>> - Receive queue=3D0x1 >>> PKT_RX_FDIR >>> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >>> length=3D1024 - nb_segs=3D2 - FDIR matched hash=3D0xc3f2 ID=3D0x5263 Un= known >>> packet type >>> - Receive queue=3D0x1 >>> PKT_RX_FDIR >>> >> >> For this issue, I think following patch can solve your problem, please a= pply this patch. >> http://dpdk.org/dev/patchwork/patch/13593/ >> > > I tried to apply it directly on 16.04 but it can't be applied. I see > it's been applied to dpdk-next-net/rel_16_07. Do you happen to have > one that would work on the latest stable 16.04 release? > > Thanks, > Dumitru > >> Beilei >> >>> Please let me know if there's more debug info that might be of interest= in >>> troubleshooting the RSS=3D0 issue. >>> >>> Thanks, >>> Dumitru >>> >>> > /Beilei >>> > >>> >> Thanks, >>> >> Dumitru >>> >> --=20 Dumitru Ceara