From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f181.google.com (mail-io0-f181.google.com [209.85.223.181]) by dpdk.org (Postfix) with ESMTP id 941584A63 for ; Fri, 22 Jul 2016 14:32:00 +0200 (CEST) Received: by mail-io0-f181.google.com with SMTP id m101so103632927ioi.2 for ; Fri, 22 Jul 2016 05:32:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=JMq77eieGB05egfrmJ3ZoiA4O5v56obeHudIwpEsn00=; b=f7zmVJcvvQWDxPV19dQtgUnDBrM2wOUDQmsljzxpXt4z9qcb4afp7h+P90fk0/JVtt LJCAr+9wFESjqqhyoLUdabsjX93VSO7muboE3Z+Tthc0M0HuFaRKJJr0nhSSWHnpAghy NTyl3A3VQjGZWyFzyfxFRQoRKb/3+u2s4o/qvGH8RokfVXJdGzYueQHEBMB6gpwP6OB2 bRa0jPBCGQcveT5+PSzlB1P1djx2/k9XSQjlQQr4OCB4MHymloHHpzaa7fTEOVtFI4jo MLz3E/NPyw2aVal9hxzYfzEwJRzorXwjbIIex3xgmgWuCdy/mWw1/z82JBQsjxGi+M23 MmTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=JMq77eieGB05egfrmJ3ZoiA4O5v56obeHudIwpEsn00=; b=AC4wnGDOzhia5cZ6mPF96+Pb59FZ6upFJA1GIzGqTZIUWHm9Y3z/2IUHAogyfPwNSN gjHy8z/2qISzX4JNYx1WevhQrur0w3t1vKsn1dCsu71unupkoHE3eyIcsQtkTdWC99Iu o1N2IrXjxpMWsJ9bExC/wK/noi/jEL7AP7RTnvn6t5R6ffjXsSPlTDRTJ5gcMySsUYSQ ziam+w4viwzz3e1A23h4YhWDU5wb2UdFkROLBtPyZujYAQ2UZMALQb/r0mMH18p6Q+hv nZhS1kSEWCl8VHE78ICOqrBiyMbhKQoBx55tB+soQrYtky0cGviROdBptqc0Z/SzFzus gRCQ== X-Gm-Message-State: AEkoouteq6x1J4lMKy27CrV0puQmgT7y3QO0zbr7h70nt6MPpllZ2hZsIgHR1eeh92ZxWjbv0wwkhTGiRSOqvA== X-Received: by 10.107.10.170 with SMTP id 42mr4623801iok.131.1469190719804; Fri, 22 Jul 2016 05:31:59 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.7.82 with HTTP; Fri, 22 Jul 2016 05:31:40 -0700 (PDT) In-Reply-To: <94479800C636CB44BD422CB454846E013B7479@SHSMSX101.ccr.corp.intel.com> References: <94479800C636CB44BD422CB454846E013B5A15@SHSMSX101.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013B5ED3@SHSMSX101.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013B7479@SHSMSX101.ccr.corp.intel.com> From: Take Ceara Date: Fri, 22 Jul 2016 14:31:40 +0200 Message-ID: To: "Xing, Beilei" Cc: "Zhang, Helin" , "Wu, Jingjing" , "dev@dpdk.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jul 2016 12:32:01 -0000 Hi Beilei, On Fri, Jul 22, 2016 at 11:04 AM, Xing, Beilei wrot= e: > Hi Ceara, > >> -----Original Message----- >> From: Take Ceara [mailto:dumitru.ceara@gmail.com] >> Sent: Thursday, July 21, 2016 6:58 PM >> To: Xing, Beilei >> Cc: Zhang, Helin ; Wu, Jingjing >> ; dev@dpdk.org >> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 >> NICs for some RX mbuf sizes >> >> >> Following your testpmd example run I managed to replicate the problem on >> my dpdk 16.04 setup like this: >> >> I have two X710 adapters connected back to back: >> $ ./tools/dpdk_nic_bind.py -s >> >> Network devices using DPDK-compatible driver >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> 0000:01:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=3Digb_uio unu= sed=3D >> 0000:81:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=3Digb_uio unu= sed=3D >> >> The firmware of the two adapters is up to date with the latest >> version: 5.04 (f5.0.40043 a1.5 n5.04 e24cd) >> >> I run testpmd with mbuf-size 1152 and txpktsize 1024 such that upon rece= ival >> the whole mbuf (except headroom) is filled. >> I enabled RX IP checksum in hw and RX RSS hashing for UDP. >> With test-pmd forward mode "rxonly" and verbose 1 I see that incoming >> packets have PKT_RX_RSS_HASH set but the hash value is 0. >> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >> --coremask=3D0xffff0 --rxq=3D16 --txq=3D16 --mbuf-size 1152 --txpkts 102= 4 -- >> enable-rx-cksum --rss-udp [...] >> testpmd> set verbose 1 >> Change verbose level from 0 to 1 >> testpmd> set fwd rxonly >> Set rxonly packet forwarding mode >> testpmd> start tx_first >> rxonly packet forwarding - CRC stripping disabled - packets/burst=3D32 >> nb forwarding cores=3D16 - nb forwarding ports=3D2 >> RX queues=3D16 - RX desc=3D128 - RX free threshold=3D32 >> RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 >> TX queues=3D16 - TX desc=3D512 - TX free threshold=3D32 >> TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 >> TX RS bit threshold=3D32 - TXQ flags=3D0xf01 port 0/queue 1: received = 32 >> packets >> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >> length=3D1024 - nb_segs=3D1 - RSS hash=3D0x0 - RSS queue=3D0x1 - (outer)= L2 >> type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP >> - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: >> Unknown - Inner L4 type: Unknown >> - Receive queue=3D0x1 >> PKT_RX_RSS_HASH >> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >> length=3D1024 - nb_segs=3D1 - RSS hash=3D0x0 - RSS queue=3D0x1 - (outer)= L2 >> type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP >> - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: >> Unknown - Inner L4 type: Unknown >> - Receive queue=3D0x1 >> PKT_RX_RSS_HASH > > What's the source ip address and destination ip address of the packet you= sent to port 0? Could you try to change ip address or port number to obser= ve if hash value changes? I remember I saw hash value was 0 before, but wit= h different ip address, there'll be different hash values. I was using the test-pmd "txonly" implementation which sends fixed UDP packets from 192.168.0.1:1024 -> 192.168.0.2:1024. I changed the test-pmd tx_only code so that it sends traffic with incremental destination IP: 192.168.0.1:1024 -> [192.168.0.2, 192.168.0.12]:1024 I also dumped the source and destination IPs in the "rxonly" pkt_burst_receive function. Then I see that packets are indeed sent to different queues but the mbuf->hash.rss value is still 0. ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i --coremask=3D0xffff0 --rxq=3D16 --txq=3D16 --mbuf-size 1152 --txpkts 1024 --enable-rx-cksum --rss-udp [...] - Receive queue=3D0xf PKT_RX_RSS_HASH src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - length=3D1024 - nb_segs=3D1 - RSS queue=3D0xa - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=3DC0A80001 DIP=3DC0A80006 - (outer) L4 type: UDP - Tunnel type: Unknown - RSS hash=3D0x0 - Inner L2 type: Unknown - RSS queue=3D0xf - RSS queue=3D0x7 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=3DC0A80001 DIP=3DC0A80007 - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown - Receive queue=3D0x7 PKT_RX_RSS_HASH src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=3DC0A80001 DIP=3DC0A80009 - type=3D0x0800 - length=3D1024 - nb_segs=3D1 - Inner L3 type: Unknown - Inne= r L4 type: Unknown - RSS hash=3D0x0 - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - RSS queue=3D0x7 - Inner L4 type: Unknown [...] testpmd> stop ------- Forward Stats for RX Port=3D 0/Queue=3D 0 -> TX Port=3D 1/Queue= =3D 0 ------- RX-packets: 0 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 1 -> TX Port=3D 1/Queue= =3D 1 ------- RX-packets: 59 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 2 -> TX Port=3D 1/Queue= =3D 2 ------- RX-packets: 48 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 3 -> TX Port=3D 1/Queue= =3D 3 ------- RX-packets: 0 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 4 -> TX Port=3D 1/Queue= =3D 4 ------- RX-packets: 59 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 5 -> TX Port=3D 1/Queue= =3D 5 ------- RX-packets: 0 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 6 -> TX Port=3D 1/Queue= =3D 6 ------- RX-packets: 0 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 7 -> TX Port=3D 1/Queue= =3D 7 ------- RX-packets: 48 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 8 -> TX Port=3D 1/Queue= =3D 8 ------- RX-packets: 0 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D 9 -> TX Port=3D 1/Queue= =3D 9 ------- RX-packets: 59 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D10 -> TX Port=3D 1/Queue= =3D10 ------- RX-packets: 48 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D11 -> TX Port=3D 1/Queue= =3D11 ------- RX-packets: 0 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D12 -> TX Port=3D 1/Queue= =3D12 ------- RX-packets: 59 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D13 -> TX Port=3D 1/Queue= =3D13 ------- RX-packets: 0 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D14 -> TX Port=3D 1/Queue= =3D14 ------- RX-packets: 0 TX-packets: 32 TX-dropped: 0 ------- Forward Stats for RX Port=3D 0/Queue=3D15 -> TX Port=3D 1/Queue= =3D15 ------- RX-packets: 48 TX-packets: 32 TX-dropped: 0 ---------------------- Forward statistics for port 0 -------------------= --- RX-packets: 428 RX-dropped: 84 RX-total: 512 TX-packets: 0 TX-dropped: 0 TX-total: 0 -------------------------------------------------------------------------= --- ---------------------- Forward statistics for port 1 -------------------= --- RX-packets: 0 RX-dropped: 0 RX-total: 0 TX-packets: 512 TX-dropped: 0 TX-total: 512 -------------------------------------------------------------------------= --- +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++= ++ RX-packets: 428 RX-dropped: 84 RX-total: 512 TX-packets: 512 TX-dropped: 0 TX-total: 512 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++= +++ I checked all the RSS hash values for all the 10 different incoming streams and they're all 0. Also, the fact that the traffic is actually distributed seems to suggest that RSS itself is working but the mbuf hash field is (I guess) either not written or corrupted. > >> >> If I use a different mbuf-size, for example 2048, everything is fine: >> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >> --coremask=3D0xffff0 --rxq=3D16 --txq=3D16 --mbuf-size 2048 --txpkts 102= 4 -- >> enable-rx-cksum --rss-udp [...] >> testpmd> set verbose 1 >> Change verbose level from 0 to 1 >> testpmd> set fwd rxonly >> Set rxonly packet forwarding mode >> testpmd> start tx_first >> rxonly packet forwarding - CRC stripping disabled - packets/burst=3D32 >> nb forwarding cores=3D16 - nb forwarding ports=3D2 >> RX queues=3D16 - RX desc=3D128 - RX free threshold=3D32 >> RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 >> TX queues=3D16 - TX desc=3D512 - TX free threshold=3D32 >> TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 >> TX RS bit threshold=3D32 - TXQ flags=3D0xf01 port 0/queue 1: received = 32 >> packets >> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >> length=3D1024 - nb_segs=3D1 - RSS hash=3D0x5263c3f2 - RSS queue=3D0x1 - >> (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) >> L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner >> L3 type: Unknown - Inner L4 type: Unknown >> - Receive queue=3D0x1 >> PKT_RX_RSS_HASH >> > > Did you send the same packet as before to port 0? > >> Another weird thing I noticed is when I run test-pmd without --enable-rx= - >> cksum (which is the default mode) then the RSS flag doesn get set at all= . >> Instead the PKT_RX_FDIR flag gets set. This happens even though fdir_mod= e >> is set to RTE_FDIR_MODE_NONE in the device >> configuration: >> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >> --coremask=3D0xffff0 --rxq=3D16 --txq=3D16 --mbuf-size 1152 --txpkts 102= 4 --rss- >> udp [...] >> testpmd> set verbose 1 >> Change verbose level from 0 to 1 >> testpmd> set fwd rxonly >> Set rxonly packet forwarding mode >> testpmd> start tx_first >> rxonly packet forwarding - CRC stripping disabled - packets/burst=3D32 >> nb forwarding cores=3D16 - nb forwarding ports=3D2 >> RX queues=3D16 - RX desc=3D128 - RX free threshold=3D32 >> RX threshold registers: pthresh=3D8 hthresh=3D8 wthresh=3D0 >> TX queues=3D16 - TX desc=3D512 - TX free threshold=3D32 >> TX threshold registers: pthresh=3D32 hthresh=3D0 wthresh=3D0 >> TX RS bit threshold=3D32 - TXQ flags=3D0xf01 port 0/queue 1: received = 16 >> packets >> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >> length=3D1024 - nb_segs=3D2 - FDIR matched hash=3D0xc3f2 ID=3D0x5263 Unk= nown >> packet type >> - Receive queue=3D0x1 >> PKT_RX_FDIR >> src=3D68:05:CA:38:6D:63 - dst=3D02:00:00:00:00:01 - type=3D0x0800 - >> length=3D1024 - nb_segs=3D2 - FDIR matched hash=3D0xc3f2 ID=3D0x5263 Unk= nown >> packet type >> - Receive queue=3D0x1 >> PKT_RX_FDIR >> > > For this issue, I think following patch can solve your problem, please ap= ply this patch. > http://dpdk.org/dev/patchwork/patch/13593/ > I tried to apply it directly on 16.04 but it can't be applied. I see it's been applied to dpdk-next-net/rel_16_07. Do you happen to have one that would work on the latest stable 16.04 release? Thanks, Dumitru > Beilei > >> Please let me know if there's more debug info that might be of interest = in >> troubleshooting the RSS=3D0 issue. >> >> Thanks, >> Dumitru >> >> > /Beilei >> > >> >> Thanks, >> >> Dumitru >> >>