From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-it0-f54.google.com (mail-it0-f54.google.com [209.85.214.54]) by dpdk.org (Postfix) with ESMTP id BF24E568E for ; Tue, 26 Jul 2016 10:57:49 +0200 (CEST) Received: by mail-it0-f54.google.com with SMTP id f6so3252082ith.0 for ; Tue, 26 Jul 2016 01:57:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=mtTaKaLSo2u6jPzrBPERuryAjpaiIiSDSVVJKkQ9tAw=; b=sKr//uK7edno8vaxzaJm2SkBVov2uqQ+chDbPqE/09woQXC+TGDfcw915S20drZP5r 9nD+ZD5u19VZhE7qNeLQKJNIomaP8XOdPFv7UIiJY+mfHpa4cpnZhChJPKHVVq3bTAbZ Def+dZ82iwCRV30+BPDd6dgK80SCZaudbeuCOdpAXjaFa9Vz5rE11ZGDzpqhmKN9p/zv s5C3BobHAXHedFM3/EO8R29u7s9nD/Y7Ni6kd7XdJpBepaQ4SHHx55jo5WGdSwvsGPKo 3iwc3k0Znm1W+Gjh1NdcYZoTdjYa/v2ZT1jcML1ux9RjA4AB7cEiXwiODbrE7lJF8QEY WpYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=mtTaKaLSo2u6jPzrBPERuryAjpaiIiSDSVVJKkQ9tAw=; b=BuqbBTwh0BEz0Cf+xQ0C/3cV7wn7j2ZomAwQLm0LQETcZlzNV10PHtY7dXTSQmO+5z ReWEJVZtOwC0Bd24MEKSLvpEp7lZT8huR/Q4UsKVA+5DQcStY61ZxqCkDHLuqo7Zkx/V y3XhtclmIXU1TMG52N0M5Di22aFPJoFs02OLM1+V42XL7nnD1SBPqmaqa+n+VzOs54Zn 17Hx2qB3eehELn0fPmFBYLgBO0J+28q6PLWfJdJTklBNXG2UMDfKIwkTqPyLhjDYaAuT Z/tuLibWAWOI780oHuSvjtlZdCRmLARSXG7PcgHqHT8Btyonp9AJcbdhtvWNGFr6yzQE sbvQ== X-Gm-Message-State: AEkoouuCl6p9S1x7hTU9fAIFzorcor892R4oBzKRbF+FVdEHxQ84s4v5DUyOtWZpnzWDy9jbX90wlvF6/oWcTg== X-Received: by 10.36.127.7 with SMTP id r7mr24897623itc.49.1469523469086; Tue, 26 Jul 2016 01:57:49 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.7.82 with HTTP; Tue, 26 Jul 2016 01:57:29 -0700 (PDT) In-Reply-To: References: <94479800C636CB44BD422CB454846E013B5A15@SHSMSX101.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013B5ED3@SHSMSX101.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013B7479@SHSMSX101.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013B81E4@SHSMSX101.ccr.corp.intel.com> From: Take Ceara Date: Tue, 26 Jul 2016 10:57:29 +0200 Message-ID: To: "Zhang, Helin" Cc: "Xing, Beilei" , "Wu, Jingjing" , "dev@dpdk.org" Content-Type: text/plain; charset=UTF-8 Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs for some RX mbuf sizes X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jul 2016 08:57:50 -0000 Hi Beilei, On Tue, Jul 26, 2016 at 10:47 AM, Zhang, Helin wrote: > Hi Ceara > > For testpmd command line, txqflags = 0xf01 should be set for receiving packets which needs more than one mbufs. > I am not sure if it is helpful for you here. Please have a try! > Just tried, and it doesn't really help: testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i --coremask=0xffff0 --rxq=2 --txq=2 --mbuf-size 1152 --txpkts 1024 --enable-rx-cksum --rss-udp --txqflags 0xf01 src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - length=1024 - nb_segs=1 - RSS hash=0x0 - RSS queue=0x0 - (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A8000A - (outer) L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - Inner L4 type: Unknown - Receive queue=0x0 PKT_RX_RSS_HASH As I was saying in my previous email the problem is that the RSS is set in the last mbuf instead of the first: http://dpdk.org/browse/dpdk/tree/drivers/net/i40e/i40e_rxtx.c#n1438 Even worse, the last rxm mbuf was already freed if it only contained the CRC which had to be stripped: http://dpdk.org/browse/dpdk/tree/drivers/net/i40e/i40e_rxtx.c#n1419 Regards, Dumitru > Regards, > Helin > >> -----Original Message----- >> From: Take Ceara [mailto:dumitru.ceara@gmail.com] >> Sent: Tuesday, July 26, 2016 4:38 PM >> To: Xing, Beilei >> Cc: Zhang, Helin ; Wu, Jingjing ; >> dev@dpdk.org >> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs >> for some RX mbuf sizes >> >> Hi Beilei, >> >> On Mon, Jul 25, 2016 at 12:04 PM, Take Ceara >> wrote: >> > Hi Beilei, >> > >> > On Mon, Jul 25, 2016 at 5:24 AM, Xing, Beilei wrote: >> >> Hi, >> >> >> >>> -----Original Message----- >> >>> From: Take Ceara [mailto:dumitru.ceara@gmail.com] >> >>> Sent: Friday, July 22, 2016 8:32 PM >> >>> To: Xing, Beilei >> >>> Cc: Zhang, Helin ; Wu, Jingjing >> >>> ; dev@dpdk.org >> >>> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for >> >>> XL710/X710 NICs for some RX mbuf sizes >> >>> >> >>> I was using the test-pmd "txonly" implementation which sends fixed >> >>> UDP packets from 192.168.0.1:1024 -> 192.168.0.2:1024. >> >>> >> >>> I changed the test-pmd tx_only code so that it sends traffic with >> >>> incremental destination IP: 192.168.0.1:1024 -> [192.168.0.2, >> >>> 192.168.0.12]:1024 >> >>> I also dumped the source and destination IPs in the "rxonly" >> >>> pkt_burst_receive function. >> >>> Then I see that packets are indeed sent to different queues but the >> >>> mbuf->hash.rss value is still 0. >> >>> >> >>> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >> >>> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024 >> >>> --enable-rx-cksum --rss-udp >> >>> >> >>> [...] >> >>> >> >>> - Receive queue=0xf >> >>> PKT_RX_RSS_HASH >> >>> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - >> >>> length=1024 - nb_segs=1 - RSS queue=0xa - (outer) L2 type: ETHER - >> >>> (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80006 - >> >>> (outer) >> >>> L4 type: UDP - Tunnel type: Unknown - RSS hash=0x0 - Inner L2 type: >> >>> Unknown - RSS queue=0xf - RSS queue=0x7 - (outer) L2 type: ETHER - >> >>> (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80007 - >> >>> (outer) >> >>> L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner >> >>> L3 type: Unknown - Inner L4 type: Unknown >> >>> - Receive queue=0x7 >> >>> PKT_RX_RSS_HASH >> >>> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - (outer) L2 type: >> >>> ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80009 >> >>> - >> >>> type=0x0800 - length=1024 - nb_segs=1 - Inner L3 type: Unknown - >> >>> Inner >> >>> L4 type: Unknown - RSS hash=0x0 - (outer) L4 type: UDP - Tunnel type: >> >>> Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - RSS >> >>> queue=0x7 - Inner L4 type: Unknown >> >>> >> >>> [...] >> >>> >> >>> testpmd> stop >> >>> ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 1/Queue= 0 >> ------- >> >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 1/Queue= 1 >> ------- >> >>> RX-packets: 59 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 1/Queue= 2 >> ------- >> >>> RX-packets: 48 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 1/Queue= 3 >> ------- >> >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 4 -> TX Port= 1/Queue= 4 >> ------- >> >>> RX-packets: 59 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 5 -> TX Port= 1/Queue= 5 >> ------- >> >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 6 -> TX Port= 1/Queue= 6 >> ------- >> >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 7 -> TX Port= 1/Queue= 7 >> ------- >> >>> RX-packets: 48 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 8 -> TX Port= 1/Queue= 8 >> ------- >> >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue= 9 -> TX Port= 1/Queue= 9 >> ------- >> >>> RX-packets: 59 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue=10 -> TX Port= 1/Queue=10 >> ------- >> >>> RX-packets: 48 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue=11 -> TX Port= 1/Queue=11 >> ------- >> >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue=12 -> TX Port= 1/Queue=12 >> ------- >> >>> RX-packets: 59 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue=13 -> TX Port= 1/Queue=13 >> ------- >> >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue=14 -> TX Port= 1/Queue=14 >> ------- >> >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 >> >>> ------- Forward Stats for RX Port= 0/Queue=15 -> TX Port= 1/Queue=15 >> ------- >> >>> RX-packets: 48 TX-packets: 32 TX-dropped: 0 >> >>> ---------------------- Forward statistics for port 0 ---------------------- >> >>> RX-packets: 428 RX-dropped: 84 RX-total: 512 >> >>> TX-packets: 0 TX-dropped: 0 TX-total: 0 >> >>> >> >>> -------------------------------------------------------------------- >> >>> -------- >> >>> >> >>> ---------------------- Forward statistics for port 1 ---------------------- >> >>> RX-packets: 0 RX-dropped: 0 RX-total: 0 >> >>> TX-packets: 512 TX-dropped: 0 TX-total: 512 >> >>> >> >>> -------------------------------------------------------------------- >> >>> -------- >> >>> >> >>> +++++++++++++++ Accumulated forward statistics for all >> >>> ports+++++++++++++++ >> >>> RX-packets: 428 RX-dropped: 84 RX-total: 512 >> >>> TX-packets: 512 TX-dropped: 0 TX-total: 512 >> >>> >> >>> >> +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >>> +++++++++++++++ >> >>> >> >>> I checked all the RSS hash values for all the 10 different incoming >> >>> streams and they're all 0. Also, the fact that the traffic is >> >>> actually distributed seems to suggest that RSS itself is working but >> >>> the mbuf hash field is (I guess) either not written or corrupted. >> >>> >> >> >> >> I tried to reproduce the problem with the same steps you used on 16.04 and >> 16.07, but I really didn't replicate it. >> >> I think you can try follow ways: >> >> 1. apply the patch I told you last time and check if the problem still exists. >> > >> > I applied the changes in the patch manually to 16.04. The RSS=0 >> > problem still exists while the FDIR issue is fixed. >> > >> >> 2. update the codebase and check if the problem still exists. >> > >> > I updated the codebase to the latest version on >> > http://dpdk.org/git/dpdk. I still see the RSS=0 issue. >> > >> >> 3. disable vector when you run testpmd, and check if the problem still exists. >> > >> > I recompiled the latest dpdk code with >> > CONFIG_RTE_LIBRTE_I40E_INC_VECTOR=n and the RSS=0 issue is still >> > there. >> > >> > My current command line is: >> > ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >> > --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024 >> > --rss-udp >> > >> > Not sure if relevant but I'm running kernel 4.2.0-27: >> > $ uname -a >> > Linux jspg2 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 >> > 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux >> > >> > Is there anything else that might help you identify the cause of the problem? >> > >> > Thanks, >> > Dumitru >> >> After some debugging in the i40e DPDK driver I figured out the problem When >> receiving packets with i40e_recv_scattered_pkts, which gets called in my case >> because the incoming packet is bigger than 1 full mbuf (the 4 bytes CRC goes in >> the second mbuf of the chain), the pkt_flags, hash, etc are set only when >> processing the last mbuf in the packet chain. However, when the hash.rss field is >> set, instead of setting it in the first mbuf of the packet it gets set in the current >> mbuf (rxm). This can also cause a lot of unpredictable behavior if the last mbuf >> only contained the CRC that was stripped as rxm would have already been freed >> by then. The line I'm referring to is: >> >> if (pkt_flags & PKT_RX_RSS_HASH) >> rxm->hash.rss = >> rte_le_to_cpu_32(rxd.wb.qword0.hi_dword.rss); >> >> I changed it to setting the rss field in first_seg instead of rxm and it works fine >> now. >> >> As far as I see this is the only place where we can receive scattered packets and >> all the other places where the RSS hash is set seem to be fine. >> Should I submit a proper patch for this or will you do it as you're more familiar to >> the code? >> >> Thanks, >> Dumitru >> >> > >> >> >> >>> > >> >>> >> >> >>> >> If I use a different mbuf-size, for example 2048, everything is fine: >> >>> >> >> >>> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >> >>> >> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 2048 --txpkts >> >>> >> 1024 >> >>> >> -- enable-rx-cksum --rss-udp [...] >> >>> >> testpmd> set verbose 1 >> >>> >> Change verbose level from 0 to 1 >> >>> >> testpmd> set fwd rxonly >> >>> >> Set rxonly packet forwarding mode >> >>> >> testpmd> start tx_first >> >>> >> rxonly packet forwarding - CRC stripping disabled - packets/burst=32 >> >>> >> nb forwarding cores=16 - nb forwarding ports=2 >> >>> >> RX queues=16 - RX desc=128 - RX free threshold=32 >> >>> >> RX threshold registers: pthresh=8 hthresh=8 wthresh=0 >> >>> >> TX queues=16 - TX desc=512 - TX free threshold=32 >> >>> >> TX threshold registers: pthresh=32 hthresh=0 wthresh=0 >> >>> >> TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: >> >>> >> received >> >>> >> 32 packets >> >>> >> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - >> >>> >> length=1024 - nb_segs=1 - RSS hash=0x5263c3f2 - RSS queue=0x1 - >> >>> >> (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - >> >>> >> (outer) >> >>> >> L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - >> >>> >> Inner >> >>> >> L3 type: Unknown - Inner L4 type: Unknown >> >>> >> - Receive queue=0x1 >> >>> >> PKT_RX_RSS_HASH >> >>> >> >> >>> > >> >>> > Did you send the same packet as before to port 0? >> >>> > >> >>> >> Another weird thing I noticed is when I run test-pmd without >> >>> >> --enable-rx- cksum (which is the default mode) then the RSS flag >> >>> >> doesn get >> >>> set at all. >> >>> >> Instead the PKT_RX_FDIR flag gets set. This happens even though >> >>> >> fdir_mode is set to RTE_FDIR_MODE_NONE in the device >> >>> >> configuration: >> >>> >> >> >>> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i >> >>> >> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts >> >>> >> 1024 >> >>> >> --rss- udp [...] >> >>> >> testpmd> set verbose 1 >> >>> >> Change verbose level from 0 to 1 >> >>> >> testpmd> set fwd rxonly >> >>> >> Set rxonly packet forwarding mode >> >>> >> testpmd> start tx_first >> >>> >> rxonly packet forwarding - CRC stripping disabled - packets/burst=32 >> >>> >> nb forwarding cores=16 - nb forwarding ports=2 >> >>> >> RX queues=16 - RX desc=128 - RX free threshold=32 >> >>> >> RX threshold registers: pthresh=8 hthresh=8 wthresh=0 >> >>> >> TX queues=16 - TX desc=512 - TX free threshold=32 >> >>> >> TX threshold registers: pthresh=32 hthresh=0 wthresh=0 >> >>> >> TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: >> >>> >> received >> >>> >> 16 packets >> >>> >> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - >> >>> >> length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 >> >>> >> Unknown packet type >> >>> >> - Receive queue=0x1 >> >>> >> PKT_RX_FDIR >> >>> >> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - >> >>> >> length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 >> >>> >> Unknown packet type >> >>> >> - Receive queue=0x1 >> >>> >> PKT_RX_FDIR >> >>> >> >> >>> > >> >>> > For this issue, I think following patch can solve your problem, >> >>> > please apply this >> >>> patch. >> >>> > http://dpdk.org/dev/patchwork/patch/13593/ >> >>> > >> >>> >> >>> I tried to apply it directly on 16.04 but it can't be applied. I see >> >>> it's been applied to dpdk-next-net/rel_16_07. Do you happen to have >> >>> one that would work on the latest stable 16.04 release? >> >> >> >> Sorry, I haven't. If there's conflict with R16.04, I think you can resolve it by >> going through the patch. >> >> >> >> Beilei >> >> >> >>> >> >>> Thanks, >> >>> Dumitru >> >>> >> >>> > Beilei >> >>> > >> >>> >> Please let me know if there's more debug info that might be of >> >>> >> interest in troubleshooting the RSS=0 issue. >> >>> >> >> >>> >> Thanks, >> >>> >> Dumitru >> >>> >> >> >>> >> > /Beilei >> >>> >> > >> >>> >> >> Thanks, >> >>> >> >> Dumitru >> >>> >> >> >> > >> > >> > >> > -- >> > Dumitru Ceara >> >> >> >> -- >> Dumitru Ceara -- Dumitru Ceara