[dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-tagged traffic

[dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-tagged traffic

[Lal, PrzemyslawX]

Hi,

I have problem with TCP Segmentation Offload of VLAN tagged traffic on i40e PMD driver - connectivity dies after trying to send first packet.
After spawning new VM non-TCP traffic (like ping) works fine. After trying to connect using SSH to the VM, connectivity is completely broken on i40e - even ping stops working. In the meanwhile connection between VMs connected to the same OVS using vhost-user ports works fine. 

This is how topology looks like (simplified):

     virtio             virtio    <-- TSO enabled on the virtio interface on virtual machine
       |                  |
    vhostuser          vhostuser  <-- TSO supported for vhost-user interfaces in DPDK since 16.04 release
       |                  |
    ovs-dpdk           ovs-dpdk   <-- VLAN tagging/untagging done in OVS-DPDK using OpenFlow rules
       |                  |
  rte_i40e_pmd       rte_i40e_pmd <-- first packet that should be offloaded breaks connectivity
       |                  |
       +----TOR switch----+

Problem occurs only with i40e NIC - when using exactly the same configuration with ixgbe NIC everything works fine, it looks like i40e PMD driver causes this problem.

Versions tested:
DPDK - 16.04, 16.07-rc3, also some post 16.07-rc2 master tested with this patch applied http://dpdk.org/ml/archives/dev/2016-July/043924.html
OVS-DPDK - recent versions from master branch with TSO RFC patch applied on top - link: http://openvswitch.org/pipermail/dev/2016-June/072871.html

In each case TSO and VLAN combo on ixgbe was working with no issues, on i40e it was always broken.

Issue was also independently reproduced by Mark Kavanagh (cc'd).

Is this known issue and is there any way to solve this issue? If not, are there any plans to fix this problem in the upcoming DPDK releases?
	
Best Regards,
Przemek

Re: [dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-tagged traffic

[Alec Istomin]

Przemek,
 Please try to reduce MTU on your virtio interfaces and see if this changes the behavior. I had to use mtu 1496  to allow tagged connectivity in my environment.  Make sure to restart OVS after MTU change.

I did not use TSO (it's not working in OVS 2.5 and dpdk 2.2.0  that i use), but the symptoms look very familiar.

Regards,
 Alec

-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Lal, PrzemyslawX
Sent: Friday, July 29, 2016 6:45
To: users@dpdk.org
Cc: Kavanagh, Mark B <mark.b.kavanagh@intel.com>; Ksiadz, MarcinX <marcinx.ksiadz@intel.com>; Siddiqui, Muhammad A <muhammad.a.siddiqui@intel.com>
Subject: [dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-tagged traffic

Hi,

I have problem with TCP Segmentation Offload of VLAN tagged traffic on i40e PMD driver - connectivity dies after trying to send first packet.
After spawning new VM non-TCP traffic (like ping) works fine. After trying to connect using SSH to the VM, connectivity is completely broken on i40e - even ping stops working. In the meanwhile connection between VMs connected to the same OVS using vhost-user ports works fine. 

This is how topology looks like (simplified):

     virtio             virtio    <-- TSO enabled on the virtio interface on virtual machine
       |                  |
    vhostuser          vhostuser  <-- TSO supported for vhost-user interfaces in DPDK since 16.04 release
       |                  |
    ovs-dpdk           ovs-dpdk   <-- VLAN tagging/untagging done in OVS-DPDK using OpenFlow rules
       |                  |
  rte_i40e_pmd       rte_i40e_pmd <-- first packet that should be offloaded breaks connectivity
       |                  |
       +----TOR switch----+

Problem occurs only with i40e NIC - when using exactly the same configuration with ixgbe NIC everything works fine, it looks like i40e PMD driver causes this problem.

Versions tested:
DPDK - 16.04, 16.07-rc3, also some post 16.07-rc2 master tested with this patch applied http://dpdk.org/ml/archives/dev/2016-July/043924.html
OVS-DPDK - recent versions from master branch with TSO RFC patch applied on top - link: http://openvswitch.org/pipermail/dev/2016-June/072871.html

In each case TSO and VLAN combo on ixgbe was working with no issues, on i40e it was always broken.

Issue was also independently reproduced by Mark Kavanagh (cc'd).

Is this known issue and is there any way to solve this issue? If not, are there any plans to fix this problem in the upcoming DPDK releases?
	
Best Regards,
Przemek

Re: [dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-tagged traffic

[Lal, PrzemyslawX]

Hi Alec,

Thanks for your reply. I know about issues with VLAN tagged traffic on i40e - I also know this workaround and tried it before posting here, but unfortunately this doesn't solve problems with TSO + VLAN. MTU is already set to 1450 on all network interfaces on the path and issue is still there.

Thanks,
Przemek

-----Original Message-----
From: Alec Istomin [mailto:A.Istomin@F5.com] 
Sent: Friday, July 29, 2016 6:55 PM
To: Lal, PrzemyslawX <przemyslawx.lal@intel.com>; users@dpdk.org
Subject: RE: [dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-tagged traffic

Przemek,
 Please try to reduce MTU on your virtio interfaces and see if this changes the behavior. I had to use mtu 1496  to allow tagged connectivity in my environment.  Make sure to restart OVS after MTU change.

I did not use TSO (it's not working in OVS 2.5 and dpdk 2.2.0  that i use), but the symptoms look very familiar.

Regards,
 Alec

-----Original Message-----
From: users [mailto:users-bounces@dpdk.org] On Behalf Of Lal, PrzemyslawX
Sent: Friday, July 29, 2016 6:45
To: users@dpdk.org
Cc: Kavanagh, Mark B <mark.b.kavanagh@intel.com>; Ksiadz, MarcinX <marcinx.ksiadz@intel.com>; Siddiqui, Muhammad A <muhammad.a.siddiqui@intel.com>
Subject: [dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-tagged traffic

Hi,

I have problem with TCP Segmentation Offload of VLAN tagged traffic on i40e PMD driver - connectivity dies after trying to send first packet.
After spawning new VM non-TCP traffic (like ping) works fine. After trying to connect using SSH to the VM, connectivity is completely broken on i40e - even ping stops working. In the meanwhile connection between VMs connected to the same OVS using vhost-user ports works fine. 

This is how topology looks like (simplified):

     virtio             virtio    <-- TSO enabled on the virtio interface on virtual machine
       |                  |
    vhostuser          vhostuser  <-- TSO supported for vhost-user interfaces in DPDK since 16.04 release
       |                  |
    ovs-dpdk           ovs-dpdk   <-- VLAN tagging/untagging done in OVS-DPDK using OpenFlow rules
       |                  |
  rte_i40e_pmd       rte_i40e_pmd <-- first packet that should be offloaded breaks connectivity
       |                  |
       +----TOR switch----+

Problem occurs only with i40e NIC - when using exactly the same configuration with ixgbe NIC everything works fine, it looks like i40e PMD driver causes this problem.

Versions tested:
DPDK - 16.04, 16.07-rc3, also some post 16.07-rc2 master tested with this patch applied http://dpdk.org/ml/archives/dev/2016-July/043924.html
OVS-DPDK - recent versions from master branch with TSO RFC patch applied on top - link: http://openvswitch.org/pipermail/dev/2016-June/072871.html

In each case TSO and VLAN combo on ixgbe was working with no issues, on i40e it was always broken.

Issue was also independently reproduced by Mark Kavanagh (cc'd).

Is this known issue and is there any way to solve this issue? If not, are there any plans to fix this problem in the upcoming DPDK releases?
	
Best Regards,
Przemek

Re: [dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-tagged traffic

[Tan, Jianfeng]

Usually, IMHO, PING works well and TCP does not, which means L4 checksum is not correctly calculated. On the receive side, you can use tcpdump to see if any packets (possible wrong L4 checksum) packets are received. Of course, you can also check statistics of the flow, if there are any packets go through.

Thanks,
Jianfeng

> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Lal,
> PrzemyslawX
> Sent: Monday, August 1, 2016 3:13 PM
> To: Alec Istomin; users@dpdk.org
> Subject: Re: [dpdk-users] i40e: issues with TCP Segmenation Offload of
> VLAN-tagged traffic
> 
> Hi Alec,
> 
> Thanks for your reply. I know about issues with VLAN tagged traffic on i40e - I
> also know this workaround and tried it before posting here, but
> unfortunately this doesn't solve problems with TSO + VLAN. MTU is already
> set to 1450 on all network interfaces on the path and issue is still there.
> 
> Thanks,
> Przemek
> 
> -----Original Message-----
> From: Alec Istomin [mailto:A.Istomin@F5.com]
> Sent: Friday, July 29, 2016 6:55 PM
> To: Lal, PrzemyslawX <przemyslawx.lal@intel.com>; users@dpdk.org
> Subject: RE: [dpdk-users] i40e: issues with TCP Segmenation Offload of
> VLAN-tagged traffic
> 
> Przemek,
>  Please try to reduce MTU on your virtio interfaces and see if this changes the
> behavior. I had to use mtu 1496  to allow tagged connectivity in my
> environment.  Make sure to restart OVS after MTU change.
> 
> I did not use TSO (it's not working in OVS 2.5 and dpdk 2.2.0  that i use), but
> the symptoms look very familiar.
> 
> Regards,
>  Alec
> 
> -----Original Message-----
> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Lal,
> PrzemyslawX
> Sent: Friday, July 29, 2016 6:45
> To: users@dpdk.org
> Cc: Kavanagh, Mark B <mark.b.kavanagh@intel.com>; Ksiadz, MarcinX
> <marcinx.ksiadz@intel.com>; Siddiqui, Muhammad A
> <muhammad.a.siddiqui@intel.com>
> Subject: [dpdk-users] i40e: issues with TCP Segmenation Offload of VLAN-
> tagged traffic
> 
> Hi,
> 
> I have problem with TCP Segmentation Offload of VLAN tagged traffic on i40e
> PMD driver - connectivity dies after trying to send first packet.
> After spawning new VM non-TCP traffic (like ping) works fine. After trying to
> connect using SSH to the VM, connectivity is completely broken on i40e -
> even ping stops working. In the meanwhile connection between VMs
> connected to the same OVS using vhost-user ports works fine.
> 
> This is how topology looks like (simplified):
> 
>      virtio             virtio    <-- TSO enabled on the virtio interface on virtual
> machine
>        |                  |
>     vhostuser          vhostuser  <-- TSO supported for vhost-user interfaces in
> DPDK since 16.04 release
>        |                  |
>     ovs-dpdk           ovs-dpdk   <-- VLAN tagging/untagging done in OVS-DPDK
> using OpenFlow rules
>        |                  |
>   rte_i40e_pmd       rte_i40e_pmd <-- first packet that should be offloaded
> breaks connectivity
>        |                  |
>        +----TOR switch----+
> 
> Problem occurs only with i40e NIC - when using exactly the same
> configuration with ixgbe NIC everything works fine, it looks like i40e PMD
> driver causes this problem.
> 
> Versions tested:
> DPDK - 16.04, 16.07-rc3, also some post 16.07-rc2 master tested with this
> patch applied http://dpdk.org/ml/archives/dev/2016-July/043924.html
> OVS-DPDK - recent versions from master branch with TSO RFC patch applied
> on top - link: http://openvswitch.org/pipermail/dev/2016-June/072871.html
> 
> In each case TSO and VLAN combo on ixgbe was working with no issues, on
> i40e it was always broken.
> 
> Issue was also independently reproduced by Mark Kavanagh (cc'd).
> 
> Is this known issue and is there any way to solve this issue? If not, are there
> any plans to fix this problem in the upcoming DPDK releases?
> 
> Best Regards,
> Przemek