From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-f54.google.com (mail-it0-f54.google.com [209.85.214.54]) by dpdk.org (Postfix) with ESMTP id E70D83989 for ; Thu, 21 Jul 2016 12:58:49 +0200 (CEST) Received: by mail-it0-f54.google.com with SMTP id f6so11165551ith.1 for ; Thu, 21 Jul 2016 03:58:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=xYskby0VXP8IGycyGE4B1+UOaRCSKATWjTUS1V4ZWTY=; b=CCQQRNMfiJWsQ1ZtWLbz1ChnZvP/4RKwsJWbCp0ssIpyxiC9i6AaFWZ26lAmwOl95k h8ggeBrgHwjbCVNUMc78kaMW9xWG0xjJ0IOJf+efvEkfryxiMckapJv40mR9J3PfybBR 3tWFbG6cvoT834pkVvZ89UTOh0EhhJKB8QKHUSFpbdRRt19jrtQdPEmNgFdf90KUPBYI JOI54n7YfnSq8mPd4lFnAy8dhT/9tAwNDaV6CQvblwvB++FVnRRJol76Pth14Faby+xy u+Jg/Om1SG/EkreJ0cYS4PC8QVr7tiPTH20GEyHqV/1kfbdBTMVhHcApsrCMuFGfw8RG smQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=xYskby0VXP8IGycyGE4B1+UOaRCSKATWjTUS1V4ZWTY=; b=JgNjL13ULtn49t8uRA471on3ZjrCAfJZKexsi/gzEK2edDBqSQ5iQozS2aNdZy1E49 TXun32MNIV1NSOaIIWnT3JJs7L1axW4VczQVeI5Eswjh44xGsRBB4hmn+fGd8ABv65EC O/WWPwPf4B6FtBa0BvwXuTksDxQg9bR9wWfv2UdQuoTu1yIOVJEyxBTIIZFAQf+PpjP8 KY9eN6WhhS/iwgw9A0winho/Z8lG95pDj1M/ajHSZtqS2l/c0SCBCD9b/UMRDcoswJtX 8UwGdm/7fVgcGhLWWwQluOHPKACAf9BpjzSBkwBgGe7HPEyOjj6pSSYsZU8tjVGYle54 Nnxg== X-Gm-Message-State: ALyK8tLJWj7lFOjzD9D31M+Q4nTam4Ay/vo8VYj87Pr/vRxnIVqNZfmBAMTOSowg9BNwW7NqfVAUdjoDyhgCVw== X-Received: by 10.36.50.206 with SMTP id j197mr61594365ita.70.1469098729249; Thu, 21 Jul 2016 03:58:49 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.7.82 with HTTP; Thu, 21 Jul 2016 03:58:29 -0700 (PDT) In-Reply-To: <94479800C636CB44BD422CB454846E013B5ED3@SHSMSX101.ccr.corp.intel.com> References: <94479800C636CB44BD422CB454846E013B5A15@SHSMSX101.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013B5ED3@SHSMSX101.ccr.corp.intel.com> From: Take Ceara Date: Thu, 21 Jul 2016 12:58:29 +0200 Message-ID: To: "Xing, Beilei" Cc: "Zhang, Helin" , "Wu, Jingjing" , "dev@dpdk.org" Content-Type: text/plain; charset=UTF-8 Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jul 2016 10:58:50 -0000 Hi Beilei, On Wed, Jul 20, 2016 at 3:59 AM, Xing, Beilei wrote: > Hi Ceara, > >> -----Original Message----- >> From: Take Ceara [mailto:dumitru.ceara@gmail.com] >> Sent: Tuesday, July 19, 2016 10:59 PM >> To: Xing, Beilei >> Cc: Zhang, Helin ; Wu, Jingjing >> ; dev@dpdk.org >> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 >> NICs for some RX mbuf sizes >> >> Hi Beilei, >> >> I changed the way I run testmpd to: >> >> testpmd -c 0x331 -w 0000:82:00.0 -w 0000:83:00.0 -- --mbuf-size 1152 --rss-ip - >> -rxq=2 --txpkts 1024 -i >> >> As far as I understand this will allocate mbufs with the same size I was using >> in my test (--mbuf-size seems to include the mbuf headroom therefore 1152 >> = 1024 + 128 headroom). >> >> testpmd> start tx_first >> io packet forwarding - CRC stripping disabled - packets/burst=32 >> nb forwarding cores=1 - nb forwarding ports=2 >> RX queues=2 - RX desc=128 - RX free threshold=32 >> RX threshold registers: pthresh=8 hthresh=8 wthresh=0 >> TX queues=1 - TX desc=512 - TX free threshold=32 >> TX threshold registers: pthresh=32 hthresh=0 wthresh=0 >> TX RS bit threshold=32 - TXQ flags=0xf01 >> testpmd> show port stats all >> >> ######################## NIC statistics for port 0 >> ######################## >> RX-packets: 18817613 RX-missed: 5 RX-bytes: 19269115888 >> RX-errors: 0 >> RX-nombuf: 0 >> TX-packets: 18818064 TX-errors: 0 TX-bytes: 19269567464 >> >> ########################################################## >> ################## >> >> ######################## NIC statistics for port 1 >> ######################## >> RX-packets: 18818392 RX-missed: 5 RX-bytes: 19269903360 >> RX-errors: 0 >> RX-nombuf: 0 >> TX-packets: 18817979 TX-errors: 0 TX-bytes: 19269479424 >> >> ########################################################## >> ################## >> >> Ttraffic is sent/received. However, I couldn't find any way to verify that the >> incoming mbufs actually have the mbuf->hash.rss field set except for starting >> test-pmd with gdb and setting a breakpoint in the io fwd engine. After doing >> that I noticed that none of the incoming packets has the PKT_RX_RSS_HASH >> flag set in ol_flags... I guess for some reason test-pmd doesn't actually >> configure RSS in this case but I fail to see where. >> > > Actually there's a way to check mbuf->hash.rss, you need set forward mode to "rxonly", and set verbose to 1. > I run testpmd with the configuration you used, and found i40e RSS works well. > With the following steps, you can see RSS hash value and receive queue, and PKT_RX_RSS_HASH is set too. > I think you can use the same way to check what you want. > > ./testpmd -c fffff -n 4 -- -i --coremask=0xffffe --rxq=16 --txq=16 --mbuf-size 1152 --rss-ip --txpkts 1024 > testpmd> set verbose 1 > testpmd> set fwd rxonly > testpmd> start > testpmd> port 0/queue 1: received 1 packets > src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - nb > - Receive queue=0x1 > PKT_RX_RSS_HASH > port 0/queue 0: received 1 packets > src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - nb_segs=1 - RSS hash=0x4e949f23 - RSS queue=0x0Unknown packet type > - Receive queue=0x0 > PKT_RX_RSS_HASH > port 0/queue 8: received 1 packets > src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - nb_segs=1 - RSS hash=0xa3c78b2b - RSS queue=0x8Unknown packet type > - Receive queue=0x8 > PKT_RX_RSS_HASH > port 0/queue 5: received 1 packets > src=00:00:01:00:0F:00 - dst=68:05:CA:32:03:4C - type=0x0800 - length=1020 - nb_segs=1 - RSS hash=0xe29b3d36 - RSS queue=0x5Unknown packet type > - Receive queue=0x5 > PKT_RX_RSS_HASH > Following your testpmd example run I managed to replicate the problem on my dpdk 16.04 setup like this: I have two X710 adapters connected back to back: $ ./tools/dpdk_nic_bind.py -s Network devices using DPDK-compatible driver ============================================ 0000:01:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=igb_uio unused= 0000:81:00.3 'Ethernet Controller X710 for 10GbE SFP+' drv=igb_uio unused= The firmware of the two adapters is up to date with the latest version: 5.04 (f5.0.40043 a1.5 n5.04 e24cd) I run testpmd with mbuf-size 1152 and txpktsize 1024 such that upon receival the whole mbuf (except headroom) is filled. I enabled RX IP checksum in hw and RX RSS hashing for UDP. With test-pmd forward mode "rxonly" and verbose 1 I see that incoming packets have PKT_RX_RSS_HASH set but the hash value is 0. ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024 --enable-rx-cksum --rss-udp [...] testpmd> set verbose 1 Change verbose level from 0 to 1 testpmd> set fwd rxonly Set rxonly packet forwarding mode testpmd> start tx_first rxonly packet forwarding - CRC stripping disabled - packets/burst=32 nb forwarding cores=16 - nb forwarding ports=2 RX queues=16 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=16 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: received 32 packets src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=1 - RSS hash=0x0 - RSS queue=0x1 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown - Receive queue=0x1 PKT_RX_RSS_HASH src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=1 - RSS hash=0x0 - RSS queue=0x1 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown - Receive queue=0x1 PKT_RX_RSS_HASH If I use a different mbuf-size, for example 2048, everything is fine: ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 2048 --txpkts 1024 --enable-rx-cksum --rss-udp [...] testpmd> set verbose 1 Change verbose level from 0 to 1 testpmd> set fwd rxonly Set rxonly packet forwarding mode testpmd> start tx_first rxonly packet forwarding - CRC stripping disabled - packets/burst=32 nb forwarding cores=16 - nb forwarding ports=2 RX queues=16 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=16 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: received 32 packets src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=1 - RSS hash=0x5263c3f2 - RSS queue=0x1 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown - Receive queue=0x1 PKT_RX_RSS_HASH Another weird thing I noticed is when I run test-pmd without --enable-rx-cksum (which is the default mode) then the RSS flag doesn get set at all. Instead the PKT_RX_FDIR flag gets set. This happens even though fdir_mode is set to RTE_FDIR_MODE_NONE in the device configuration: ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024 --rss-udp [...] testpmd> set verbose 1 Change verbose level from 0 to 1 testpmd> set fwd rxonly Set rxonly packet forwarding mode testpmd> start tx_first rxonly packet forwarding - CRC stripping disabled - packets/burst=32 nb forwarding cores=16 - nb forwarding ports=2 RX queues=16 - RX desc=128 - RX free threshold=32 RX threshold registers: pthresh=8 hthresh=8 wthresh=0 TX queues=16 - TX desc=512 - TX free threshold=32 TX threshold registers: pthresh=32 hthresh=0 wthresh=0 TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: received 16 packets src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 Unknown packet type - Receive queue=0x1 PKT_RX_FDIR src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 Unknown packet type - Receive queue=0x1 PKT_RX_FDIR Please let me know if there's more debug info that might be of interest in troubleshooting the RSS=0 issue. Thanks, Dumitru > /Beilei > >> Thanks, >> Dumitru >>