[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Franck BAUDIN]

Hello,

This is a repost of https://lists.01.org/pipermail/dpdk-ovs/2014-September/001788.html as the issue might be related to dpdk.org vhost code.

I am using dpdk-ovs 1.1.0 (latest release) as follow :

linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --- Niantic --- > linux-native

[*] commit 8fd8bebc051704d7caecfed8d8a065a79c022329
Author: Adrien Mazarguil <adrien.mazarguil@6wind.com>
Date:   Mon Sep 1 12:31:11 2014 +0200

UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest are incorrect, as showed with tcpdump on linux-native.

Is it supposed to work? If yes, what is the proper qemu version to use and its parameters?

I am using qemu version provided by dpdk-ovs (same issue with PDDK 1.7.0 and latest one) and I followed this documentation: https://github.com/01org/dpdk-ovs/blob/development/docs/04_Sample_Configurations/02_Userspace-vHost.md

Thanks!
Franck



This message and any attachments (the "message") are confidential, intended solely for the addressees. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this message from your system. In this case, you are not authorized to use, copy this message and/or disclose the content to any other person. E-mails are susceptible to alteration. Neither Qosmos nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.

Ce message et toutes ses pièces jointes (ci-après le "message")sont confidentiels et établis à l'intention exclusive de ses destinataires. Si vous avez reçu ce message par erreur, merci d’en informer immédiatement son émetteur par courrier électronique et d’effacer ce message de votre système. Dans cette hypothèse, vous n’êtes pas autorisé à utiliser, copier ce message et/ou en divulguer le contenu à un tiers. Tout message électronique est susceptible d'altération. Qosmos et ses filiales déclinent toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié.

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Thomas Monjalon]

Hi Franck,

2014-09-02 13:20, Franck BAUDIN:
> I am using dpdk-ovs 1.1.0 (latest release) as follow :
> 
> linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --- Niantic --- > linux-native
> 
[...]
> UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest are incorrect, as showed with tcpdump on linux-native.

Thanks for reporting.
Could you try virtio without vhost?


> This message and any attachments (the "message") are confidential, intended solely for the addressees. If you are not the intended recipient, please notify the sender immediately by e-mail and delete this message from your system. In this case, you are not authorized to use, copy this message and/or disclose the content to any other person. E-mails are susceptible to alteration. Neither Qosmos nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified.
> 
> Ce message et toutes ses pièces jointes (ci-après le "message")sont confidentiels et établis à l'intention exclusive de ses destinataires. Si vous avez reçu ce message par erreur, merci d’en informer immédiatement son émetteur par courrier électronique et d’effacer ce message de votre système. Dans cette hypothèse, vous n’êtes pas autorisé à utiliser, copier ce message et/ou en divulguer le contenu à un tiers. Tout message électronique est susceptible d'altération. Qosmos et ses filiales déclinent toute responsabilité au titre de ce message s'il a été altéré, déformé ou falsifié.

This big disclaimer block has no sense on a public mailing list.
Please try to remove it.

Thanks
-- 
Thomas

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Franck Baudin]

Hi Thomas,

On 09/02/14 15:29, Thomas Monjalon wrote:
> UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the guest are incorrect, as showed with tcpdump on linux-native.
> Thanks for reporting.
> Could you try virtio without vhost?
Nope, dpdk-ovs supports only vhost.

Thanks!
Franck

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Gray, Mark D]

> 
> Hi Franck,
> 
> 2014-09-02 13:20, Franck BAUDIN:
> > I am using dpdk-ovs 1.1.0 (latest release) as follow :
> >
> > linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --
> - Niantic --- > linux-native
> >
> [...]
> > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the
> guest are incorrect, as showed with tcpdump on linux-native.
> 
> Thanks for reporting.
> Could you try virtio without vhost?
> 
> 

Looping in the dpdk-ovs list. 

* Does the new vhost API allow a user to know if all the relevant offloads have been
turned on/off for that interface? It seems that this is possible through the virtio_net
structure but it would be good to get some feedback from the relevant person
working on DPDK (Huawei?).

* If this is the case, then it is probably in the realm of the vswitch do the actual 
checksum (for VM-VM) or correctly configure the NIC when sending out through
the physical interface.

Comments?

Mark

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Xie, Huawei]

> -----Original Message-----
> From: Gray, Mark D
> Sent: Wednesday, September 03, 2014 6:01 PM
> To: Thomas Monjalon; Franck BAUDIN; Xie, Huawei
> Cc: dev@dpdk.org; dpdk-ovs@lists.01.org
> Subject: RE: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest
> (virtIO/vhost)
> 
> >
> > Hi Franck,
> >
> > 2014-09-02 13:20, Franck BAUDIN:
> > > I am using dpdk-ovs 1.1.0 (latest release) as follow :
> > >
> > > linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest DPDK [*]) < --
> > - Niantic --- > linux-native
> > >
> > [...]
> > > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by the
> > guest are incorrect, as showed with tcpdump on linux-native.
> >
> > Thanks for reporting.
> > Could you try virtio without vhost?
> >
> >
> 
> Looping in the dpdk-ovs list.
> 
> * Does the new vhost API allow a user to know if all the relevant offloads have
> been
> turned on/off for that interface? It seems that this is possible through the
> virtio_net
> structure but it would be good to get some feedback from the relevant person
> working on DPDK (Huawei?).
> 
> * If this is the case, then it is probably in the realm of the vswitch do the actual
> checksum (for VM-VM) or correctly configure the NIC when sending out through
> the physical interface.
> 
> Comments?


Mark:
So far not supported. This is important as well in VxLan case. For the packet flow  
Guest A->  virtio -> ..->OVDK->.. -> Guest B.
1) If guest A and B are on different host machines, say A and B respectively,  and if the nic on A supports
vxlan checksum offload, then both guest and host needn't generate checksum, the nic will  
generate checksum for both inner and outer packet.
2) In VM2VM case, as it is trusted communication channel, could we negotiate with the guest tcp stack not to verify checksum 
for received packet? 

> 
> Mark

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Franck Baudin]

Hi,

On 09/03/14 13:13, Xie, Huawei wrote:
> Looping in the dpdk-ovs list.
>
> * Does the new vhost API allow a user to know if all the relevant offloads have
> been
> turned on/off for that interface? It seems that this is possible through the
> virtio_net
> structure but it would be good to get some feedback from the relevant person
> working on DPDK (Huawei?).
>
> * If this is the case, then it is probably in the realm of the vswitch do the actual
> checksum (for VM-VM) or correctly configure the NIC when sending out through
> the physical interface.
>
> Comments?
>
> Mark:
> So far not supported. This is important as well in VxLan case. For the packet flow
> Guest A->  virtio -> ..->OVDK->.. -> Guest B.
> 1) If guest A and B are on different host machines, say A and B respectively,  and if the nic on A supports
> vxlan checksum offload, then both guest and host needn't generate checksum, the nic will
> generate checksum for both inner and outer packet.
> 2) In VM2VM case, as it is trusted communication channel, could we negotiate with the guest tcp stack not to verify checksum
> for received packet?
The problem is that any TCP packet send by a vanilla Linux guest through 
vhost is incorrect (VM to anything, including other colocalied VMs). In 
other words, the VM cannot use TCP. QEMU options and ethtool -K csum off 
tso off ("TCP stack negociation") have no effect, maybe because the 
vhost backend is misbehaving.

Franck

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Gray, Mark D]

> 
> Hi,
> 
> On 09/03/14 13:13, Xie, Huawei wrote:
> > Looping in the dpdk-ovs list.
> >
> > * Does the new vhost API allow a user to know if all the relevant
> > offloads have been turned on/off for that interface? It seems that
> > this is possible through the virtio_net structure but it would be good
> > to get some feedback from the relevant person working on DPDK
> > (Huawei?).
> >
> > * If this is the case, then it is probably in the realm of the vswitch
> > do the actual checksum (for VM-VM) or correctly configure the NIC when
> > sending out through the physical interface.
> >
> > Comments?
> >
> > Mark:
> > So far not supported. This is important as well in VxLan case. For the
> > packet flow Guest A->  virtio -> ..->OVDK->.. -> Guest B.
> > 1) If guest A and B are on different host machines, say A and B
> > respectively,  and if the nic on A supports vxlan checksum offload,
> > then both guest and host needn't generate checksum, the nic will generate
> checksum for both inner and outer packet.
> > 2) In VM2VM case, as it is trusted communication channel, could we
> > negotiate with the guest tcp stack not to verify checksum for received
> packet?
> The problem is that any TCP packet send by a vanilla Linux guest through
> vhost is incorrect (VM to anything, including other colocalied VMs). In other
> words, the VM cannot use TCP. QEMU options and ethtool -K csum off tso
> off ("TCP stack negociation") have no effect, maybe because the vhost
> backend is misbehaving.

This sounds like a DPDK issue. However, we did "clone and own" the vhost
code as it is a sample app. It may have changed since we integrated it.

We plan to use the vhost library interface when it is available on dpdk.org. So
that would pick up the latest version of the vhost code.

> 
> Franck

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Gray, Mark D]

> > >
> > > Hi Franck,
> > >
> > > 2014-09-02 13:20, Franck BAUDIN:
> > > > I am using dpdk-ovs 1.1.0 (latest release) as follow :
> > > >
> > > > linux-guest (no DPKD) <---  virtIO ---> OVDK 1.1.0 (with latest
> > > > DPDK [*]) < --
> > > - Niantic --- > linux-native
> > > >
> > > [...]
> > > > UDP/ICMP connectivity is fine, but TCP checksum of packet sent by
> > > > the
> > > guest are incorrect, as showed with tcpdump on linux-native.
> > >
> > > Thanks for reporting.
> > > Could you try virtio without vhost?
> > >
> > >
> >
> > Looping in the dpdk-ovs list.
> >
> > * Does the new vhost API allow a user to know if all the relevant
> > offloads have been turned on/off for that interface? It seems that
> > this is possible through the virtio_net structure but it would be good
> > to get some feedback from the relevant person working on DPDK
> > (Huawei?).
> >
> > * If this is the case, then it is probably in the realm of the vswitch
> > do the actual checksum (for VM-VM) or correctly configure the NIC when
> > sending out through the physical interface.
> >
> > Comments?
> 
> 
> Mark:
> So far not supported. This is important as well in VxLan case. For the packet

Ok 

> flow Guest A->  virtio -> ..->OVDK->.. -> Guest B.
> 1) If guest A and B are on different host machines, say A and B respectively,
> and if the nic on A supports vxlan checksum offload, then both guest and
> host needn't generate checksum, the nic will generate checksum for both
> inner and outer packet.

The vswitch would need to support this

> 2) In VM2VM case, as it is trusted communication channel, could we
> negotiate with the guest tcp stack not to verify checksum for received
> packet?

Perhaps

> 
> >
> > Mark

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Xie, Huawei]

> -----Original Message-----
> From: Franck Baudin [mailto:franck.baudin@qosmos.com]
> Sent: Wednesday, September 03, 2014 10:13 PM
> To: Xie, Huawei; Gray, Mark D; Thomas Monjalon
> Cc: dev@dpdk.org; dpdk-ovs@lists.01.org
> Subject: Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest
> (virtIO/vhost)
> 
> Hi,
> 
> On 09/03/14 13:13, Xie, Huawei wrote:
> > Looping in the dpdk-ovs list.
> >
> > * Does the new vhost API allow a user to know if all the relevant offloads have
> > been
> > turned on/off for that interface? It seems that this is possible through the
> > virtio_net
> > structure but it would be good to get some feedback from the relevant person
> > working on DPDK (Huawei?).
> >
> > * If this is the case, then it is probably in the realm of the vswitch do the actual
> > checksum (for VM-VM) or correctly configure the NIC when sending out
> through
> > the physical interface.
> >
> > Comments?
> >
> > Mark:
> > So far not supported. This is important as well in VxLan case. For the packet
> flow
> > Guest A->  virtio -> ..->OVDK->.. -> Guest B.
> > 1) If guest A and B are on different host machines, say A and B respectively,
> and if the nic on A supports
> > vxlan checksum offload, then both guest and host needn't generate checksum,
> the nic will
> > generate checksum for both inner and outer packet.
> > 2) In VM2VM case, as it is trusted communication channel, could we negotiate
> with the guest tcp stack not to verify checksum
> > for received packet?
> The problem is that any TCP packet send by a vanilla Linux guest through
> vhost is incorrect (VM to anything, including other colocalied VMs). In
> other words, the VM cannot use TCP. QEMU options and ethtool -K csum off
> tso off ("TCP stack negociation") have no effect, maybe because the
> vhost backend is misbehaving.
> 
> Franck

Hi Franck:
I checked your original thread.
root at linux-native:~# tcpdump -vnei eth0
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto TCP (6), length 60)
   1.1.1.3.34272 > 1.1.1.2.5555: Flags [S], cksum 0x0435 (incorrect -> 0xb2dd), seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr 0,nop,wscale 7], length 0
17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto TCP (6), length 40)
    1.1.1.2.5555 > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq 0, ack 1963818357, win 0, length 0

The packet from the guest, received on the native machine, has "correct" checksum 0x1db4. The TCP connection is refused is due to this packet has R flag. Could you check if it is the firewall that blocks the 
connection?

Besides, I did ssh tcp test between two guest VMs,  the ssh connection could be established correctly with tx checksum off. I didn't use OVDK, but set the arp table and route table manually.
The packet flow is
	Guest A(virtio)<->vhost example->Guest B(virtio)

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Franck Baudin]

Hi Huawei,

On 09/04/14 04:54, Xie, Huawei wrote:
> Hi Franck:
> I checked your original thread.
> root at linux-native:~# tcpdump -vnei eth0
> tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
> 17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto TCP (6), length 60)
>     1.1.1.3.34272 > 1.1.1.2.5555: Flags [S], cksum 0x0435 (incorrect -> 0xb2dd), seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr 0,nop,wscale 7], length 0
> 17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto TCP (6), length 40)
>      1.1.1.2.5555 > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq 0, ack 1963818357, win 0, length 0
>
> The packet from the guest, received on the native machine, has "correct" checksum 0x1db4. The TCP connection is refused is due to this packet has R flag. Could you check if it is the firewall that blocks the
> connection?
>
> Besides, I did ssh tcp test between two guest VMs,  the ssh connection could be established correctly with tx checksum off. I didn't use OVDK, but set the arp table and route table manually.
> The packet flow is
> 	Guest A(virtio)<->vhost example->Guest B(virtio)
I re-made a test between two identical VMs, UDP and ICMP are fine. Some 
observations:
on both client & server:

ethtool -k eth1 | grep ": on"
generic-receive-offload: on
highdma: on [fixed]
rx-vlan-filter: on [fixed]

ethtool -k eth1
Features for eth1:
rx-checksumming: off [fixed]
tx-checksumming: off
         tx-checksum-ipv4: off [fixed]
         tx-checksum-unneeded: off [fixed]
         tx-checksum-ip-generic: off [fixed]
         tx-checksum-ipv6: off [fixed]
         tx-checksum-fcoe-crc: off [fixed]
         tx-checksum-sctp: off [fixed]
scatter-gather: off
         tx-scatter-gather: off [fixed]
         tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: off
         tx-tcp-segmentation: off [fixed]
         tx-tcp-ecn-segmentation: off [fixed]
         tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: off [requested on]
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]

On the client side:
11:00:23.296592 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], 
proto TCP
(6), length 60)
     1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0x0433 (incorrect ->
0xe3bd), seq 3885781956, win 14600, options [mss 1460,sackOK,TS val 
4294914387
ecr 0,nop,wscale 5], length 0
         0x0000:  4500 003c 6297 4000 4006 d420 0101 0102 E..<b.@.@.......
         0x0010:  0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S.....
         0x0020:  a002 3908 0433 0000 0204 05b4 0402 080a ..9..3..........
         0x0030:  ffff 3153 0000 0000 0103 0305 ..1S........
11:00:23.296881 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF], 
proto TCP
(6), length 40)
     1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0,
ack 3885781957, win 0, length 0
         0x0000:  4500 0028 198f 4000 4006 1d3d 0101 0101 E..(..@.@..=....
         0x0010:  0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S.
         0x0020:  5014 0000 b5e6 0000                      P.......

On the server side:
11:00:23.631991 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF], 
proto TCP
(6), length 60)
     1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0xe3bd (correct), seq
3885781956, win 14600, options [mss 1460,sackOK,TS val 4294914387 ecr
0,nop,wscale 5], length 0
         0x0000:  4500 003c 6297 4000 4006 d420 0101 0102 E..<b.@.@.......
         0x0010:  0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S.....
         0x0020:  a002 3908 e3bd 0000 0204 05b4 0402 080a ..9.............
         0x0030:  ffff 3153 0000 0000 0103 0305 ..1S........
11:00:23.632034 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF], 
proto TCP
(6), length 40)
     1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0,
ack 3885781957, win 0, length 0
         0x0000:  4500 0028 198f 4000 4006 1d3d 0101 0101 E..(..@.@..=....
         0x0010:  0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S.
         0x0020:  5014 0000 b5e6 0000                      P.......


On the client, ethtool announce that TX checksum offload is disabled, 
but tcpdump show it's not. On the server side, all packets looks fine 
from a checksum perspective but the SYN is answered by a RST. Please 
note that the same images behave correctly with a regular openvswitch, 
as long as TX checksuming offloading is disabled.

If anyone has an equivalent setup, functioning properly, thanks for 
sharing its configurations parameters. if not, I might be able to wait 
for vhost backend PMD driver: I will then with testpmd instead of 
dpdk-ovs, to see where is the problem (maybe in my configuration...). 
When so you plan to release the vhost backend PMD driver?

Thanks!
Franck

Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest (virtIO/vhost)

[Xie, Huawei]

> -----Original Message-----
> From: Franck Baudin [mailto:franck.baudin@qosmos.com]
> Sent: Thursday, September 04, 2014 5:24 PM
> To: Xie, Huawei; Gray, Mark D; Thomas Monjalon
> Cc: dev@dpdk.org; dpdk-ovs@lists.01.org
> Subject: Re: [dpdk-dev] Wrong TCP checksum of packets sent by Linux guest
> (virtIO/vhost)
> 
> Hi Huawei,
> 
> On 09/04/14 04:54, Xie, Huawei wrote:
> > Hi Franck:
> > I checked your original thread.
> > root at linux-native:~# tcpdump -vnei eth0
> > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535
> bytes
> > 17:27:09.262926 00:1b:21:b9:9b:2c > 52:54:00:51:51:12, ethertype IPv4
> (0x0800), length 74: (tos 0x0, ttl 64, id 47743, offset 0, flags [DF], proto TCP (6),
> length 60)
> >     1.1.1.3.34272 > 1.1.1.2.5555: Flags [S], cksum 0x0435 (incorrect -> 0xb2dd),
> seq 1963818356, win 14600, options [mss 1460,sackOK,TS val 49530635 ecr
> 0,nop,wscale 7], length 0
> > 17:27:09.263220 52:54:00:51:51:12 > 00:1b:21:b9:9b:2c, ethertype IPv4
> (0x0800), length 60: (tos 0x0, ttl 64, id 3367, offset 0, flags [DF], proto TCP (6),
> length 40)
> >      1.1.1.2.5555 > 1.1.1.3.34272: Flags [R.], cksum 0x1db4 (correct), seq 0, ack
> 1963818357, win 0, length 0
> >
> > The packet from the guest, received on the native machine, has "correct"
> checksum 0x1db4. The TCP connection is refused is due to this packet has R flag.
> Could you check if it is the firewall that blocks the
> > connection?
> >
> > Besides, I did ssh tcp test between two guest VMs,  the ssh connection could
> be established correctly with tx checksum off. I didn't use OVDK, but set the arp
> table and route table manually.
> > The packet flow is
> > 	Guest A(virtio)<->vhost example->Guest B(virtio)
> I re-made a test between two identical VMs, UDP and ICMP are fine. Some
> observations:
> on both client & server:
> 
> ethtool -k eth1 | grep ": on"
> generic-receive-offload: on
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> 
> ethtool -k eth1
> Features for eth1:
> rx-checksumming: off [fixed]
> tx-checksumming: off
>          tx-checksum-ipv4: off [fixed]
>          tx-checksum-unneeded: off [fixed]
>          tx-checksum-ip-generic: off [fixed]
>          tx-checksum-ipv6: off [fixed]
>          tx-checksum-fcoe-crc: off [fixed]
>          tx-checksum-sctp: off [fixed]
> scatter-gather: off
>          tx-scatter-gather: off [fixed]
>          tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: off
>          tx-tcp-segmentation: off [fixed]
>          tx-tcp-ecn-segmentation: off [fixed]
>          tx-tcp6-segmentation: off [fixed]
> udp-fragmentation-offload: off [fixed]
> generic-segmentation-offload: off [requested on]
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: off [fixed]
> tx-fcoe-segmentation: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> 
> On the client side:
> 11:00:23.296592 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF],
> proto TCP
> (6), length 60)
>      1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0x0433 (incorrect ->
> 0xe3bd), seq 3885781956, win 14600, options [mss 1460,sackOK,TS val
> 4294914387
> ecr 0,nop,wscale 5], length 0
>          0x0000:  4500 003c 6297 4000 4006 d420 0101 0102 E..<b.@.@.......
>          0x0010:  0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S.....
>          0x0020:  a002 3908 0433 0000 0204 05b4 0402 080a ..9..3..........
>          0x0030:  ffff 3153 0000 0000 0103 0305 ..1S........
> 11:00:23.296881 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF],
> proto TCP
> (6), length 40)
>      1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0,
> ack 3885781957, win 0, length 0
>          0x0000:  4500 0028 198f 4000 4006 1d3d 0101 0101 E..(..@.@..=....
>          0x0010:  0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S.
>          0x0020:  5014 0000 b5e6 0000                      P.......
> 
> On the server side:
> 11:00:23.631991 IP (tos 0x0, ttl 64, id 25239, offset 0, flags [DF],
> proto TCP
> (6), length 60)
>      1.1.1.2.47192 > 1.1.1.1.dsf: Flags [S], cksum 0xe3bd (correct), seq
> 3885781956, win 14600, options [mss 1460,sackOK,TS val 4294914387 ecr
> 0,nop,wscale 5], length 0
>          0x0000:  4500 003c 6297 4000 4006 d420 0101 0102 E..<b.@.@.......
>          0x0010:  0101 0101 b858 022b e79c 53c4 0000 0000 .....X.+..S.....
>          0x0020:  a002 3908 e3bd 0000 0204 05b4 0402 080a ..9.............
>          0x0030:  ffff 3153 0000 0000 0103 0305 ..1S........
> 11:00:23.632034 IP (tos 0x0, ttl 64, id 6543, offset 0, flags [DF],
> proto TCP
> (6), length 40)
>      1.1.1.1.dsf > 1.1.1.2.47192: Flags [R.], cksum 0xb5e6 (correct), seq 0,
> ack 3885781957, win 0, length 0
>          0x0000:  4500 0028 198f 4000 4006 1d3d 0101 0101 E..(..@.@..=....
>          0x0010:  0101 0102 022b b858 0000 0000 e79c 53c5 .....+.X......S.
>          0x0020:  5014 0000 b5e6 0000                      P.......
> 
> 
> On the client, ethtool announce that TX checksum offload is disabled,
> but tcpdump show it's not. On the server side, all packets looks fine
> from a checksum perspective but the SYN is answered by a RST. Please
> note that the same images behave correctly with a regular openvswitch,
> as long as TX checksuming offloading is disabled.
In your previous test case, everything looks fine except that server sends a RST 
packet.
In this test, weird the virtio client sends packet with incorrect checksum with tx 
checksum offloading off. But at least RX side always receives correct checksum 
which indicates the virtio driver on client side generates checksum for it .

The server receives correct SYN packet with correct checksum, but it sends the RST packet, 
better root cause this issue first.

With vhost example as the backend, nc tests or ssh tests always works fine on my system.
If you have bandwidth, you could also test vhost backend. Vhost command line fyi:
dpdk/examples/vhost/build/vhost-switch -c 3 -n 4 --huge-dir /dev/hugepages --socket-mem=512,0 
-- -p 1 --rx-retry 1 --zero-copy 0 --vm2vm 2

Btw, I don't set any csum...=off in qemu command line.
> 
> If anyone has an equivalent setup, functioning properly, thanks for
> sharing its configurations parameters. if not, I might be able to wait
> for vhost backend PMD driver: I will then with testpmd instead of
> dpdk-ovs, to see where is the problem (maybe in my configuration...).
> When so you plan to release the vhost backend PMD driver?
> 
There is no vhost backend PMD driver but a vhost library for integration with OVDK.

> Thanks!
> Franck