From: "Xie, Huawei" <huawei.xie@intel.com>
To: Franck Baudin <franck.baudin@qosmos.com>,
"Gray, Mark D" <mark.d.gray@intel.com>,
Thomas Monjalon <thomas.monjalon@6wind.com>
Cc: "dev@dpdk.org" <dev@dpdk.org>,
"dpdk-ovs@lists.01.org" <dpdk-ovs@lists.01.org>
Subject: Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)
Date: Thu, 4 Sep 2014 10:35:05 +0000 [thread overview]
Message-ID: <C37D651A908B024F974696C65296B57B0F286002@SHSMSX101.ccr.corp.intel.com> (raw)
In-Reply-To: <54082FC0.1060205@qosmos.com>
> -----Original Message-----
> From: Franck Baudin [mailto:franck.baudin@qosmos.com]
> Sent: Thursday, September 04, 2014 5:24 PM
> To: Xie, Huawei; Gray, Mark D; Thomas Monjalon
> Cc: dev@dpdk.org; dpdk-ovs@lists.01.org
> Subject: Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest
> (virtIO/vhost)
>
> Hi Huawei,
>
> On 09/04/14 04:54, Xie, Huawei wrote:
> > Hi Franck:
> > I checked your original thread.
> > root at linux-native:~# tcpdump -vnei eth0
> > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535
> bytes
> > 17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto TCP (6),
> length 60)
> > 1.1.1.3.34272 > 1.1.1.2.5555: Flags [S], cksum 0x0435 (incorrect -> 0xb2dd),
> seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr
> 0,nop,wscale 7], length 0
> > 17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto TCP (6),
> length 40)
> > 1.1.1.2.5555 > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq 0, ack
> 1963818357, win 0, length 0
> >
> > The packet from the guest, received on the native machine, has "correct"
> checksum 0x1db4. The TCP connection is refused is due to this packet has R flag.
> Could you check if it is the firewall that blocks the
> > connection?
> >
> > Besides, I did ssh tcp test between two guest VMs, the ssh connection could
> be established correctly with tx checksum off. I didn't use OVDK, but set the arp
> table and route table manually.
> > The packet flow is
> > Guest A(virtio)<->vhost example->Guest B(virtio)
> I re-made a test between two identical VMs, UDP and ICMP are fine. Some
> observations:
> on both client & server:
>
> ethtool -k eth1 | grep ": on"
> generic-receive-offload: on
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
>
> ethtool -k eth1
> Features for eth1:
> rx-checksumming: off [fixed]
> tx-checksumming: off
> tx-checksum-ipv4: off [fixed]
> tx-checksum-unneeded: off [fixed]
> tx-checksum-ip-generic: off [fixed]
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: off
> tx-scatter-gather: off [fixed]
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: off
> tx-tcp-segmentation: off [fixed]
> tx-tcp-ecn-segmentation: off [fixed]
> tx-tcp6-segmentation: off [fixed]
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: off [requested on]
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
>
> On the client side:
> 11:00:23.296592 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF],
> proto TCP
> (6), length 60)
> 1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0x0433 (incorrect ->
> 0xe3bd), seq 3885781956, win 14600, options [mss 1460,sackOK,TS val
> 4294914387
> ecr 0,nop,wscale 5], length 0
> 0x0000: 4500 003c 6297 4000 4006 d420 0101 0102 E..<b.@.@.......
> 0x0010: 0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S.....
> 0x0020: a002 3908 0433 0000 0204 05b4 0402 080a ..9..3..........
> 0x0030: ffff 3153 0000 0000 0103 0305 ..1S........
> 11:00:23.296881 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF],
> proto TCP
> (6), length 40)
> 1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0,
> ack 3885781957, win 0, length 0
> 0x0000: 4500 0028 198f 4000 4006 1d3d 0101 0101 E..(..@.@..=....
> 0x0010: 0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S.
> 0x0020: 5014 0000 b5e6 0000 P.......
>
> On the server side:
> 11:00:23.631991 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF],
> proto TCP
> (6), length 60)
> 1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0xe3bd (correct), seq
> 3885781956, win 14600, options [mss 1460,sackOK,TS val 4294914387 ecr
> 0,nop,wscale 5], length 0
> 0x0000: 4500 003c 6297 4000 4006 d420 0101 0102 E..<b.@.@.......
> 0x0010: 0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S.....
> 0x0020: a002 3908 e3bd 0000 0204 05b4 0402 080a ..9.............
> 0x0030: ffff 3153 0000 0000 0103 0305 ..1S........
> 11:00:23.632034 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF],
> proto TCP
> (6), length 40)
> 1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0,
> ack 3885781957, win 0, length 0
> 0x0000: 4500 0028 198f 4000 4006 1d3d 0101 0101 E..(..@.@..=....
> 0x0010: 0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S.
> 0x0020: 5014 0000 b5e6 0000 P.......
>
>
> On the client, ethtool announce that TX checksum offload is disabled,
> but tcpdump show it's not. On the server side, all packets looks fine
> from a checksum perspective but the SYN is answered by a RST. Please
> note that the same images behave correctly with a regular openvswitch,
> as long as TX checksuming offloading is disabled.
In your previous test case, everything looks fine except that server sends a RST
packet.
In this test, weird the virtio client sends packet with incorrect checksum with tx
checksum offloading off. But at least RX side always receives correct checksum
which indicates the virtio driver on client side generates checksum for it .
The server receives correct SYN packet with correct checksum, but it sends the RST packet,
better root cause this issue first.
With vhost example as the backend, nc tests or ssh tests always works fine on my system.
If you have bandwidth, you could also test vhost backend. Vhost command line fyi:
dpdk/examples/vhost/build/vhost-switch -c 3 -n 4 --huge-dir /dev/hugepages --socket-mem=512,0
-- -p 1 --rx-retry 1 --zero-copy 0 --vm2vm 2
Btw, I don't set any csum...=off in qemu command line.
>
> If anyone has an equivalent setup, functioning properly, thanks for
> sharing its configurations parameters. if not, I might be able to wait
> for vhost backend PMD driver: I will then with testpmd instead of
> dpdk-ovs, to see where is the problem (maybe in my configuration...).
> When so you plan to release the vhost backend PMD driver?
>
There is no vhost backend PMD driver but a vhost library for integration with OVDK.
> Thanks!
> Franck
next prev parent reply other threads:[~2014-09-04 10:30 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-09-02 13:20 Franck BAUDIN
2014-09-02 13:29 ` Thomas Monjalon
2014-09-02 14:37 ` Franck Baudin
2014-09-03 10:00 ` Gray, Mark D
2014-09-03 11:13 ` Xie, Huawei
2014-09-03 14:12 ` Franck Baudin
2014-09-03 16:07 ` Gray, Mark D
2014-09-04 2:54 ` Xie, Huawei
2014-09-04 9:24 ` Franck Baudin
2014-09-04 10:35 ` Xie, Huawei [this message]
2014-09-03 16:15 ` Gray, Mark D
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=C37D651A908B024F974696C65296B57B0F286002@SHSMSX101.ccr.corp.intel.com \
--to=huawei.xie@intel.com \
--cc=dev@dpdk.org \
--cc=dpdk-ovs@lists.01.org \
--cc=franck.baudin@qosmos.com \
--cc=mark.d.gray@intel.com \
--cc=thomas.monjalon@6wind.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).