From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f170.google.com (mail-io0-f170.google.com [209.85.223.170]) by dpdk.org (Postfix) with ESMTP id 8A48B1E34 for ; Fri, 5 May 2017 15:05:22 +0200 (CEST) Received: by mail-io0-f170.google.com with SMTP id k91so7608003ioi.1 for ; Fri, 05 May 2017 06:05:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infinite-io.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=maIltB9poBMtFuWOJ8Af0eSMeZAX0uxaYLnPhcY6BF8=; b=g8AR9hFNZZnv2wJGxy7mazTEkGKdpyoiZeOeWzRbGZaqyZ20KJtx9D9o2m2M4i5Sdl 7d1VfRAzJ8oY6NNd4BwqbYlyPw0BJ0Bte92itplCbV6DZK+RQ9TPKOOn5u1XVKyl2yVC lfxUXmM0CIT06LjPgJZpw6YgYOP4G8bLYIFJn//60goHcj3Mk7IECytcCFXscFHs7bl6 J1mFCJi8Z6l/KwPpRDBhBDpt/jl7dr5xk8yYTRHHq1/jTcanivSOwbY6k/2+iuY87Wjl 2gR0dZGN+Z6A2zhy5G37MBRW0bOOboTIrx8oHT7/BOp0GNkPlgMsA3KvSfNnN2FDyCjW nnFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=maIltB9poBMtFuWOJ8Af0eSMeZAX0uxaYLnPhcY6BF8=; b=PJMkHZEkLZ2A5U0aj//SSJF6ELA2cXSEHnOo+yAVgKM4Sth6RCD6XGQimizlz+jTpB yALontrtrtNhHNL61yBMgB7/hxfBhl5fqkWsKxHCYrsdhaDiQ9uR6OPenKKLGCcAg4fK cLnYVw8ziVyWrZBEHfTpPKYDwI59kNgstE+dxkauc+GG7Muc1eB4JETaPwlXFVxFSEcD ylf2v6XFerObJ8nSXC/ASLAVK5l0EyMoLK2Lp3hhws9t0o966N+rM/nKOJvZN2sNa4ZU RJFo7s/Yowu0QjiiPsz8xw458oxTFWg/pmQdoF7CPp0LtZZOKtPDhuZEwpvOX86uSw33 NTKA== X-Gm-Message-State: AN3rC/6fHRH3wNT0AzhqbxzHHkTHoSfWgu4Cux6iZP5Mw2EB5fWZWqMF IDcns0kmKD1LzApNI60OFoDkhaDxrmqO X-Received: by 10.107.58.135 with SMTP id h129mr43793331ioa.70.1493989521216; Fri, 05 May 2017 06:05:21 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.44.23 with HTTP; Fri, 5 May 2017 06:05:20 -0700 (PDT) In-Reply-To: References: <7B7539B0-8CB0-4DDB-B329-D11F295A2604@intel.com> From: Matt Laswell Date: Fri, 5 May 2017 08:05:20 -0500 Message-ID: To: "dev@dpdk.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] Occasional instability in RSS Hashes/Queues from X540 NIC X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 May 2017 13:05:23 -0000 On Thu, May 4, 2017 at 1:15 PM, Matt Laswell wrote: > Hey Keith, > > Here is a hexdump of a subset of one of my packet captures. In this > capture, all of the packets are part of the same TCP connection, which > happens to be NFSv3 traffic. All of them except packet number 6 get the > correct RSS hash and go to the right queue. Packet number 6 (an NFS rename > reply with an NFS error) gets RSS hash 0 and goes to queue 0. Whenever I > repeat this test, the reply to this particular rename attempt always goes > to the wrong core, though it seemingly differs from the rest of the flow > only in layers 4-7. > > I'll also attach a pcap to this email, in case that's a more convenient > way to interact with the packets. > > -- > Matt Laswell > laswell@infinite.io > > > 16:08:37.093306 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags [P.], > seq 3173509264:3173509380, ack 3244259549, win 580, options [nop,nop,TS val > 23060466 ecr 490971270], length 116: NFS request xid 2690728524 112 access > fh Unknown/8B6BFEBB04000000CFABD10301000000FFFFFFFF00000000DABC050201000000 > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ > ACCESS_EXTEND|NFS_ACCESS_DELETE > 0x0000: 4500 00a8 6d0f 4000 4006 b121 0a97 0351 E...m.@.@..!...Q > 0x0010: 0a97 03a1 029b 0801 bd27 e890 c15f 78dd .........'..._x. > 0x0020: 8018 0244 1cba 0000 0101 080a 015f dff2 ...D........._.. > 0x0030: 1d43 a086 8000 0070 a061 424c 0000 0000 .C.....p.aBL.... > 0x0040: 0000 0002 0001 86a3 0000 0003 0000 0004 ................ > 0x0050: 0000 0001 0000 0020 0107 8d2f 0000 0007 .........../.... > 0x0060: 6573 7869 3275 3100 0000 0000 0000 0000 esxi2u1......... > 0x0070: 0000 0001 0000 0000 0000 0000 0000 0000 ................ > 0x0080: 0000 0020 8b6b febb 0400 0000 cfab d103 .....k.......... > 0x0090: 0100 0000 ffff ffff 0000 0000 dabc 0502 ................ > 0x00a0: 0100 0000 0000 001f ........ > 16:08:37.095837 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags [P.], > seq 1:125, ack 116, win 28688, options [nop,nop,TS val 490971270 ecr > 23060466], length 124: NFS reply xid 2690728524 reply ok 120 access c 001f > 0x0000: 4500 00b0 1b80 4000 4006 02a9 0a97 03a1 E.....@.@....... > 0x0010: 0a97 0351 0801 029b c15f 78dd bd27 e904 ...Q....._x..'.. > 0x0020: 8018 7010 a61a 0000 0101 080a 1d43 a086 ..p..........C.. > 0x0030: 015f dff2 8000 0078 a061 424c 0000 0001 ._.....x.aBL.... > 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 0x0050: 0000 0000 0000 0001 0000 0002 0000 01ed ................ > 0x0060: 0000 0003 0000 0000 0000 0000 0000 0000 ................ > 0x0070: 0000 0029 0000 0000 0000 0800 0000 00ff ...)............ > 0x0080: ffff 00ff 0000 0000 bbfe 6b8b 0000 0001 ..........k..... > 0x0090: 03d1 abcf 5908 f554 3272 e4e6 5908 f554 ....Y..T2r..Y..T > 0x00a0: 3272 e4e6 5908 f554 3365 2612 0000 001f 2r..Y..T3e&..... > 16:08:37.096235 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags [P.], > seq 256:372, ack 285, win 589, options [nop,nop,TS val 23060467 ecr > 490971270], length 116: NFS request xid 2724282956 112 access fh Unknown/ > 8B6BFEBB04000000D0ABD10301000000FFFFFFFF00000000DABC050201000000 > NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_ > ACCESS_EXTEND|NFS_ACCESS_DELETE > 0x0000: 4500 00a8 6d11 4000 4006 b11f 0a97 0351 E...m.@.@......Q > 0x0010: 0a97 03a1 029b 0801 bd27 e990 c15f 79f9 .........'..._y. > 0x0020: 8018 024d 1cba 0000 0101 080a 015f dff3 ...M........._.. > 0x0030: 1d43 a086 8000 0070 a261 424c 0000 0000 .C.....p.aBL.... > 0x0040: 0000 0002 0001 86a3 0000 0003 0000 0004 ................ > 0x0050: 0000 0001 0000 0020 0107 8d2f 0000 0007 .........../.... > 0x0060: 6573 7869 3275 3100 0000 0000 0000 0000 esxi2u1......... > 0x0070: 0000 0001 0000 0000 0000 0000 0000 0000 ................ > 0x0080: 0000 0020 8b6b febb 0400 0000 d0ab d103 .....k.......... > 0x0090: 0100 0000 ffff ffff 0000 0000 dabc 0502 ................ > 0x00a0: 0100 0000 0000 001f ........ > 16:08:37.098361 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags [P.], > seq 285:409, ack 372, win 28688, options [nop,nop,TS val 490971270 ecr > 23060467], length 124: NFS reply xid 2724282956 reply ok 120 access c 001f > 0x0000: 4500 00b0 1b81 4000 4006 02a8 0a97 03a1 E.....@.@....... > 0x0010: 0a97 0351 0801 029b c15f 79f9 bd27 ea04 ...Q....._y..'.. > 0x0020: 8018 7010 ec45 0000 0101 080a 1d43 a086 ..p..E.......C.. > 0x0030: 015f dff3 8000 0078 a261 424c 0000 0001 ._.....x.aBL.... > 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 0x0050: 0000 0000 0000 0001 0000 0002 0000 01ed ................ > 0x0060: 0000 0004 0000 0000 0000 0000 0000 0000 ................ > 0x0070: 0000 0050 0000 0000 0000 0800 0000 00ff ...P............ > 0x0080: ffff 00ff 0000 0000 bbfe 6b8b 0000 0001 ..........k..... > 0x0090: 03d1 abd0 5908 f554 3536 88ea 5908 f554 ....Y..T56..Y..T > 0x00a0: 3536 88ea 5908 f555 01ff bf76 0000 001f 56..Y..U...v.... > 16:08:37.099013 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags [P.], > seq 652:856, ack 813, win 605, options [nop,nop,TS val 23060467 ecr > 490971270], length 204: NFS request xid 2774614604 200 rename fh Unknown/ > 8B6BFEBB04000000D0ABD10301000000FFFFFFFF00000000DABC050201000000 > "DirReplaceNotEmpty_ovr" -> fh Unknown/8B6BFEBB04000000D0ABD103010000 > 00FFFFFFFF00000000DABC050201000000 "DirReplaceNotEmpty_src" > 0x0000: 4500 0100 6d14 4000 4006 b0c4 0a97 0351 E...m.@.@......Q > 0x0010: 0a97 03a1 029b 0801 bd27 eb1c c15f 7c09 .........'..._|. > 0x0020: 8018 025d 1d12 0000 0101 080a 015f dff3 ...]........._.. > 0x0030: 1d43 a086 8000 00c8 a561 424c 0000 0000 .C.......aBL.... > 0x0040: 0000 0002 0001 86a3 0000 0003 0000 000e ................ > 0x0050: 0000 0001 0000 0020 0107 8d2f 0000 0007 .........../.... > 0x0060: 6573 7869 3275 3100 0000 0000 0000 0000 esxi2u1......... > 0x0070: 0000 0001 0000 0000 0000 0000 0000 0000 ................ > 0x0080: 0000 0020 8b6b febb 0400 0000 d0ab d103 .....k.......... > 0x0090: 0100 0000 ffff ffff 0000 0000 dabc 0502 ................ > 0x00a0: 0100 0000 0000 0016 4469 7252 6570 6c61 ........DirRepla > 0x00b0: 6365 4e6f 7445 6d70 7479 5f6f 7672 0000 ceNotEmpty_ovr.. > 0x00c0: 0000 0020 8b6b febb 0400 0000 d0ab d103 .....k.......... > 0x00d0: 0100 0000 ffff ffff 0000 0000 dabc 0502 ................ > 0x00e0: 0100 0000 0000 0016 4469 7252 6570 6c61 ........DirRepla > 0x00f0: 6365 4e6f 7445 6d70 7479 5f73 7263 0000 ceNotEmpty_src.. > 16:08:37.101770 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags [P.], > seq 4294966865:4294966961, ack 4294967244, win 28688, options [nop,nop,TS > val 490971270 ecr 23060467], length 96: NFS reply xid 2774614604 reply ok > 92 rename ERROR: File exists > 0x0000: 4500 0094 1b82 4000 4006 02c3 0a97 03a1 E.....@.@....... > 0x0010: 0a97 0351 0801 029b c15f 772d bd27 e85c ...Q....._w-.'.\ > 0x0020: 8018 7010 c0f8 0000 0101 080a 1d43 a086 ..p..........C.. > 0x0030: 015f dff3 8000 005c a561 424c 0000 0001 ._.....\.aBL.... > 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 0x0050: 0000 0011 0000 0001 0000 0000 0000 0050 ...............P > 0x0060: 5908 f554 3536 88ea 5908 f555 01ff bf76 Y..T56..Y..U...v > 0x0070: 0000 0000 0000 0001 0000 0000 0000 0050 ...............P > 0x0080: 5908 f554 3536 88ea 5908 f555 01ff bf76 Y..T56..Y..U...v > 0x0090: 0000 0000 .... > 16:08:37.101774 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags [.], > ack 813, win 605, options [nop,nop,TS val 23060468 ecr > 490971270,nop,nop,sack 1 {4294966865:4294966961}], length 0 > 0x0000: 4500 0040 6d15 4000 4006 b183 0a97 0351 E..@m.@.@......Q > 0x0010: 0a97 03a1 029b 0801 bd27 ebe8 c15f 7c09 .........'..._|. > 0x0020: b010 025d 1c52 0000 0101 080a 015f dff4 ...].R......._.. > 0x0030: 1d43 a086 0101 050a c15f 772d c15f 778d .C......._w-._w. > > On Thu, May 4, 2017 at 11:34 AM, Wiles, Keith > wrote: > >> >> > On May 4, 2017, at 8:04 AM, Matt Laswell wrote: >> > >> > Hey Folks, >> > >> > I'm seeing some strange behavior with regard to the RSS hash values in >> my >> > applications and was hoping somebody might have some pointers on where >> to >> > look. In my application, I'm using RSS to divide work among multiple >> > cores, each of which services a single RX queue. When dealing with a >> > single long-lived TCP connection, I occasionally see packets going to >> the >> > wrong core. That is, almost all of the packets in the connection go to >> > core 5 in this case, but every once in a while, one goes to core 0 >> instead. >> > >> > Upon further investigation, I find two problems are occurring. The >> first >> > is that problem packets have the RSS hash value in the mbuf incorrectly >> set >> > to zero. They are therefore put in queue zero, where they are read by >> core >> > zero. Other packets from the same connection that occur immediately >> before >> > and after the packet in question have the correct hash value and >> therefore >> > go to a different core. The second problem is that we sometimes see >> > packets in which the RSS hash in the mbuf appears correct, but the >> packets >> > are incorrectly put into queue zero. As with the first, this results in >> > the wrong core getting the packet. Either one of these confuses the >> state >> > tracking we're doing per-core. >> > >> > A few details: >> > >> > - Using an Intel X540-AT2 NIC and the igb_uio driver >> > - DPDK 16.04 >> > - A particular packet in our workflow always encounters this problem. >> > - Retransmissions of the packet in question also encounter the problem >> > - The packet is IPv4, with header length of 20 (so no options), no >> > fragmentation. >> > - The only differences I can see in the IP header between packets that >> > get the right hash value and those that get the wrong one are in the >> IP ID, >> > total length, and checksum fields. >> > - Using ETH_RSS_IPV4 >> > - The packet is TCP with about 100 bytes of payload - it's not a jumbo >> > or a runt >> > - We fill the key in with 0x6d5a to get symmetric hashing of both >> sides >> > of the connection >> > - We only configure RSS information at boot; things like the key or >> > header fields are not being changed dynamically >> > - Traffic load is light when the problem occurs >> > >> > Is anybody aware of an errata, either in the NIC or the PMD's >> configuration >> > of it that might explain something like this? Failing that, if you ran >> > into this sort of behavior, how would you approach finding the reason >> for >> > the error? Every failure mode I can think of would tend to affect all >> of >> > the packets in the connection consistently, even if incorrectly. >> >> Just to add more information to this email, can you provide hexdumps of >> the packets to help someone maybe spot the problem? >> >> Need the previous OK packet plus the one after it and the failing packets >> you are seeing. >> >> I do not know why this is happening as I do not know of any errata to >> explain this issue. >> >> > >> > Thanks in advance for any ideas. >> > >> > -- >> > Matt Laswell >> > laswell@infinite.io >> >> Regards, >> Keith >> >> > It appears that the list stripped off the pcap attachment that I included in the above email. If anybody is interested in looking at it, please let me know and I'd be happy to send it off-list. -- Matt Laswell laswell@infinite.io