From: Matt Laswell <laswell@infinite.io>
To: "dev@dpdk.org" <dev@dpdk.org>
Subject: Re: [dpdk-dev] Occasional instability in RSS Hashes/Queues from X540 NIC
Date: Fri, 5 May 2017 08:05:20 -0500 [thread overview]
Message-ID: <CA+GnqAow09XA8MxiD+PDhZAKi6q4v0zOihKHcR9mdk9W+ryofg@mail.gmail.com> (raw)
In-Reply-To: <CA+GnqAppBUiSO=pfpJz6vVBNyov+zPo7rWzirLVnh7MjnPTKvw@mail.gmail.com>
On Thu, May 4, 2017 at 1:15 PM, Matt Laswell <laswell@infinite.io> wrote:
> Hey Keith,
>
> Here is a hexdump of a subset of one of my packet captures. In this
> capture, all of the packets are part of the same TCP connection, which
> happens to be NFSv3 traffic. All of them except packet number 6 get the
> correct RSS hash and go to the right queue. Packet number 6 (an NFS rename
> reply with an NFS error) gets RSS hash 0 and goes to queue 0. Whenever I
> repeat this test, the reply to this particular rename attempt always goes
> to the wrong core, though it seemingly differs from the rest of the flow
> only in layers 4-7.
>
> I'll also attach a pcap to this email, in case that's a more convenient
> way to interact with the packets.
>
> --
> Matt Laswell
> laswell@infinite.io
>
>
> 16:08:37.093306 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags [P.],
> seq 3173509264:3173509380, ack 3244259549, win 580, options [nop,nop,TS val
> 23060466 ecr 490971270], length 116: NFS request xid 2690728524 112 access
> fh Unknown/8B6BFEBB04000000CFABD10301000000FFFFFFFF00000000DABC050201000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_
> ACCESS_EXTEND|NFS_ACCESS_DELETE
> 0x0000: 4500 00a8 6d0f 4000 4006 b121 0a97 0351 E...m.@.@..!...Q
> 0x0010: 0a97 03a1 029b 0801 bd27 e890 c15f 78dd .........'..._x.
> 0x0020: 8018 0244 1cba 0000 0101 080a 015f dff2 ...D........._..
> 0x0030: 1d43 a086 8000 0070 a061 424c 0000 0000 .C.....p.aBL....
> 0x0040: 0000 0002 0001 86a3 0000 0003 0000 0004 ................
> 0x0050: 0000 0001 0000 0020 0107 8d2f 0000 0007 .........../....
> 0x0060: 6573 7869 3275 3100 0000 0000 0000 0000 esxi2u1.........
> 0x0070: 0000 0001 0000 0000 0000 0000 0000 0000 ................
> 0x0080: 0000 0020 8b6b febb 0400 0000 cfab d103 .....k..........
> 0x0090: 0100 0000 ffff ffff 0000 0000 dabc 0502 ................
> 0x00a0: 0100 0000 0000 001f ........
> 16:08:37.095837 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags [P.],
> seq 1:125, ack 116, win 28688, options [nop,nop,TS val 490971270 ecr
> 23060466], length 124: NFS reply xid 2690728524 reply ok 120 access c 001f
> 0x0000: 4500 00b0 1b80 4000 4006 02a9 0a97 03a1 E.....@.@.......
> 0x0010: 0a97 0351 0801 029b c15f 78dd bd27 e904 ...Q....._x..'..
> 0x0020: 8018 7010 a61a 0000 0101 080a 1d43 a086 ..p..........C..
> 0x0030: 015f dff2 8000 0078 a061 424c 0000 0001 ._.....x.aBL....
> 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 0x0050: 0000 0000 0000 0001 0000 0002 0000 01ed ................
> 0x0060: 0000 0003 0000 0000 0000 0000 0000 0000 ................
> 0x0070: 0000 0029 0000 0000 0000 0800 0000 00ff ...)............
> 0x0080: ffff 00ff 0000 0000 bbfe 6b8b 0000 0001 ..........k.....
> 0x0090: 03d1 abcf 5908 f554 3272 e4e6 5908 f554 ....Y..T2r..Y..T
> 0x00a0: 3272 e4e6 5908 f554 3365 2612 0000 001f 2r..Y..T3e&.....
> 16:08:37.096235 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags [P.],
> seq 256:372, ack 285, win 589, options [nop,nop,TS val 23060467 ecr
> 490971270], length 116: NFS request xid 2724282956 112 access fh Unknown/
> 8B6BFEBB04000000D0ABD10301000000FFFFFFFF00000000DABC050201000000
> NFS_ACCESS_READ|NFS_ACCESS_LOOKUP|NFS_ACCESS_MODIFY|NFS_
> ACCESS_EXTEND|NFS_ACCESS_DELETE
> 0x0000: 4500 00a8 6d11 4000 4006 b11f 0a97 0351 E...m.@.@......Q
> 0x0010: 0a97 03a1 029b 0801 bd27 e990 c15f 79f9 .........'..._y.
> 0x0020: 8018 024d 1cba 0000 0101 080a 015f dff3 ...M........._..
> 0x0030: 1d43 a086 8000 0070 a261 424c 0000 0000 .C.....p.aBL....
> 0x0040: 0000 0002 0001 86a3 0000 0003 0000 0004 ................
> 0x0050: 0000 0001 0000 0020 0107 8d2f 0000 0007 .........../....
> 0x0060: 6573 7869 3275 3100 0000 0000 0000 0000 esxi2u1.........
> 0x0070: 0000 0001 0000 0000 0000 0000 0000 0000 ................
> 0x0080: 0000 0020 8b6b febb 0400 0000 d0ab d103 .....k..........
> 0x0090: 0100 0000 ffff ffff 0000 0000 dabc 0502 ................
> 0x00a0: 0100 0000 0000 001f ........
> 16:08:37.098361 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags [P.],
> seq 285:409, ack 372, win 28688, options [nop,nop,TS val 490971270 ecr
> 23060467], length 124: NFS reply xid 2724282956 reply ok 120 access c 001f
> 0x0000: 4500 00b0 1b81 4000 4006 02a8 0a97 03a1 E.....@.@.......
> 0x0010: 0a97 0351 0801 029b c15f 79f9 bd27 ea04 ...Q....._y..'..
> 0x0020: 8018 7010 ec45 0000 0101 080a 1d43 a086 ..p..E.......C..
> 0x0030: 015f dff3 8000 0078 a261 424c 0000 0001 ._.....x.aBL....
> 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 0x0050: 0000 0000 0000 0001 0000 0002 0000 01ed ................
> 0x0060: 0000 0004 0000 0000 0000 0000 0000 0000 ................
> 0x0070: 0000 0050 0000 0000 0000 0800 0000 00ff ...P............
> 0x0080: ffff 00ff 0000 0000 bbfe 6b8b 0000 0001 ..........k.....
> 0x0090: 03d1 abd0 5908 f554 3536 88ea 5908 f554 ....Y..T56..Y..T
> 0x00a0: 3536 88ea 5908 f555 01ff bf76 0000 001f 56..Y..U...v....
> 16:08:37.099013 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags [P.],
> seq 652:856, ack 813, win 605, options [nop,nop,TS val 23060467 ecr
> 490971270], length 204: NFS request xid 2774614604 200 rename fh Unknown/
> 8B6BFEBB04000000D0ABD10301000000FFFFFFFF00000000DABC050201000000
> "DirReplaceNotEmpty_ovr" -> fh Unknown/8B6BFEBB04000000D0ABD103010000
> 00FFFFFFFF00000000DABC050201000000 "DirReplaceNotEmpty_src"
> 0x0000: 4500 0100 6d14 4000 4006 b0c4 0a97 0351 E...m.@.@......Q
> 0x0010: 0a97 03a1 029b 0801 bd27 eb1c c15f 7c09 .........'..._|.
> 0x0020: 8018 025d 1d12 0000 0101 080a 015f dff3 ...]........._..
> 0x0030: 1d43 a086 8000 00c8 a561 424c 0000 0000 .C.......aBL....
> 0x0040: 0000 0002 0001 86a3 0000 0003 0000 000e ................
> 0x0050: 0000 0001 0000 0020 0107 8d2f 0000 0007 .........../....
> 0x0060: 6573 7869 3275 3100 0000 0000 0000 0000 esxi2u1.........
> 0x0070: 0000 0001 0000 0000 0000 0000 0000 0000 ................
> 0x0080: 0000 0020 8b6b febb 0400 0000 d0ab d103 .....k..........
> 0x0090: 0100 0000 ffff ffff 0000 0000 dabc 0502 ................
> 0x00a0: 0100 0000 0000 0016 4469 7252 6570 6c61 ........DirRepla
> 0x00b0: 6365 4e6f 7445 6d70 7479 5f6f 7672 0000 ceNotEmpty_ovr..
> 0x00c0: 0000 0020 8b6b febb 0400 0000 d0ab d103 .....k..........
> 0x00d0: 0100 0000 ffff ffff 0000 0000 dabc 0502 ................
> 0x00e0: 0100 0000 0000 0016 4469 7252 6570 6c61 ........DirRepla
> 0x00f0: 6365 4e6f 7445 6d70 7479 5f73 7263 0000 ceNotEmpty_src..
> 16:08:37.101770 IP 10.151.3.161.nfsd > 10.151.3.81.disclose: Flags [P.],
> seq 4294966865:4294966961, ack 4294967244, win 28688, options [nop,nop,TS
> val 490971270 ecr 23060467], length 96: NFS reply xid 2774614604 reply ok
> 92 rename ERROR: File exists
> 0x0000: 4500 0094 1b82 4000 4006 02c3 0a97 03a1 E.....@.@.......
> 0x0010: 0a97 0351 0801 029b c15f 772d bd27 e85c ...Q....._w-.'.\
> 0x0020: 8018 7010 c0f8 0000 0101 080a 1d43 a086 ..p..........C..
> 0x0030: 015f dff3 8000 005c a561 424c 0000 0001 ._.....\.aBL....
> 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000 ................
> 0x0050: 0000 0011 0000 0001 0000 0000 0000 0050 ...............P
> 0x0060: 5908 f554 3536 88ea 5908 f555 01ff bf76 Y..T56..Y..U...v
> 0x0070: 0000 0000 0000 0001 0000 0000 0000 0050 ...............P
> 0x0080: 5908 f554 3536 88ea 5908 f555 01ff bf76 Y..T56..Y..U...v
> 0x0090: 0000 0000 ....
> 16:08:37.101774 IP 10.151.3.81.disclose > 10.151.3.161.nfsd: Flags [.],
> ack 813, win 605, options [nop,nop,TS val 23060468 ecr
> 490971270,nop,nop,sack 1 {4294966865:4294966961}], length 0
> 0x0000: 4500 0040 6d15 4000 4006 b183 0a97 0351 E..@m.@.@......Q
> 0x0010: 0a97 03a1 029b 0801 bd27 ebe8 c15f 7c09 .........'..._|.
> 0x0020: b010 025d 1c52 0000 0101 080a 015f dff4 ...].R......._..
> 0x0030: 1d43 a086 0101 050a c15f 772d c15f 778d .C......._w-._w.
>
> On Thu, May 4, 2017 at 11:34 AM, Wiles, Keith <keith.wiles@intel.com>
> wrote:
>
>>
>> > On May 4, 2017, at 8:04 AM, Matt Laswell <laswell@infinite.io> wrote:
>> >
>> > Hey Folks,
>> >
>> > I'm seeing some strange behavior with regard to the RSS hash values in
>> my
>> > applications and was hoping somebody might have some pointers on where
>> to
>> > look. In my application, I'm using RSS to divide work among multiple
>> > cores, each of which services a single RX queue. When dealing with a
>> > single long-lived TCP connection, I occasionally see packets going to
>> the
>> > wrong core. That is, almost all of the packets in the connection go to
>> > core 5 in this case, but every once in a while, one goes to core 0
>> instead.
>> >
>> > Upon further investigation, I find two problems are occurring. The
>> first
>> > is that problem packets have the RSS hash value in the mbuf incorrectly
>> set
>> > to zero. They are therefore put in queue zero, where they are read by
>> core
>> > zero. Other packets from the same connection that occur immediately
>> before
>> > and after the packet in question have the correct hash value and
>> therefore
>> > go to a different core. The second problem is that we sometimes see
>> > packets in which the RSS hash in the mbuf appears correct, but the
>> packets
>> > are incorrectly put into queue zero. As with the first, this results in
>> > the wrong core getting the packet. Either one of these confuses the
>> state
>> > tracking we're doing per-core.
>> >
>> > A few details:
>> >
>> > - Using an Intel X540-AT2 NIC and the igb_uio driver
>> > - DPDK 16.04
>> > - A particular packet in our workflow always encounters this problem.
>> > - Retransmissions of the packet in question also encounter the problem
>> > - The packet is IPv4, with header length of 20 (so no options), no
>> > fragmentation.
>> > - The only differences I can see in the IP header between packets that
>> > get the right hash value and those that get the wrong one are in the
>> IP ID,
>> > total length, and checksum fields.
>> > - Using ETH_RSS_IPV4
>> > - The packet is TCP with about 100 bytes of payload - it's not a jumbo
>> > or a runt
>> > - We fill the key in with 0x6d5a to get symmetric hashing of both
>> sides
>> > of the connection
>> > - We only configure RSS information at boot; things like the key or
>> > header fields are not being changed dynamically
>> > - Traffic load is light when the problem occurs
>> >
>> > Is anybody aware of an errata, either in the NIC or the PMD's
>> configuration
>> > of it that might explain something like this? Failing that, if you ran
>> > into this sort of behavior, how would you approach finding the reason
>> for
>> > the error? Every failure mode I can think of would tend to affect all
>> of
>> > the packets in the connection consistently, even if incorrectly.
>>
>> Just to add more information to this email, can you provide hexdumps of
>> the packets to help someone maybe spot the problem?
>>
>> Need the previous OK packet plus the one after it and the failing packets
>> you are seeing.
>>
>> I do not know why this is happening as I do not know of any errata to
>> explain this issue.
>>
>> >
>> > Thanks in advance for any ideas.
>> >
>> > --
>> > Matt Laswell
>> > laswell@infinite.io
>>
>> Regards,
>> Keith
>>
>>
>
It appears that the list stripped off the pcap attachment that I included
in the above email. If anybody is interested in looking at it, please let
me know and I'd be happy to send it off-list.
--
Matt Laswell
laswell@infinite.io
next prev parent reply other threads:[~2017-05-05 13:05 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-04 13:04 Matt Laswell
2017-05-04 16:34 ` Wiles, Keith
2017-05-04 18:15 ` Matt Laswell
2017-05-05 13:05 ` Matt Laswell [this message]
2017-05-08 8:55 ` Yang, Qiming
2017-07-18 8:58 ` Yang, Qiming
2017-07-18 13:43 ` Matt Laswell
2017-07-18 14:01 ` Yuanhan Liu
2017-07-19 6:17 ` Li, Xiaoyun
2017-07-19 8:13 ` Li, Xiaoyun
2017-07-19 10:54 ` Yuanhan Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CA+GnqAow09XA8MxiD+PDhZAKi6q4v0zOihKHcR9mdk9W+ryofg@mail.gmail.com \
--to=laswell@infinite.io \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).