Re: [dpdk-dev] [vpp-dev] VLAN packets dropped... ?

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

 
> I suppose this has to do with what is expected usage of the PKT_RX_VLAN_PKT offload flag. Is it set only for VLAN packets with the
> VLAN stripped or should always be set if VLAN is/was present in the received packet. It seems that different DPDK drivers are
> behaving differently which will make it really hard for VPP to take advantage of NIC and driver offload capability to provide better
> performance.

Yes, that's true ixgbe/igb from one side and i40e do raise PKT_RX_VLAN_PKT in a different manner.
There is an attempt to make it unified across all supported devices:
 http://dpdk.org/dev/patchwork/patch/12938/

Though, I am not sure it will help you with your issue.
As I understand, for you the desired behaviour is:
If it is a vlan packet, keep the packet intact (don't strip the vlan) and raise PKT_RX_VLAN_PK inside mbuf->ol_flags.
That's what ixgbe is doing with rte_eth_conf.rxmode.hw_vlan_strip==0.
Correct?
As far as I know, i40e HW doesn't provide such ability.
i40e Receive HW Descriptor can only flag was VLAN tag stripped from the packet or not,
but if stripping is disabled it wouldn't indicate in any way is VLAN tag is present inside the packet or not.
I am CC-ing it to dpdk.org in case I am missing something here.
Probably someone knows a way to make it work in that way.    
Konstantin

> 
> If VPP cannot rely on offload flags for VLAN so packets always have to go through ethernet-input node, there is a performance cost.
> For the 10GE case, before the inverse patch of the ixgbe driver, 10GE Rx-vector path removed support of offload flag with DPDK 16.04
> and so ethernet-input node is always used. The 10GE IPv4 throughput rate dropped from 6.17MPPSx2 to 4.92MPPSx2 for bidirectional
> traffic (2 streams each with a single IP address as destination) on a single core on my server. Konstantin suggested at that time to use
> scalar mode which does support offload flags properly. The scalar mode did by-pass ethernet-input and provided throughput of
> 5.72MPPS. We ended up inverse patched the ixgbe driver to restore vector mode offload flag support as the original restriction (the
> reason offload flag support was removed) would not affect VPP.
> 
> I think for 40GE driver to provide offload flag support in vector mode but not give indication of presence of VLAN tag is just wrong. This
> make the offload flag support useless for VPP.
> 
> Regards,
> John
> 
> -----Original Message-----
> From: Ananyev, Konstantin [mailto:konstantin.ananyev@intel.com]
> Sent: Wednesday, May 25, 2016 11:30 AM
> To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan
> Cc: vpp-dev; Zhang, Helin
> Subject: RE: [vpp-dev] VLAN packets dropped... ?
> 
> 
> >
> > I see what you are getting at, Konstantin. The VPP init code does not
> > enable VLAN strip for Intel NICs as VLAN tag must be in the packet for
> > sub-interface lookup by ethernet-input node. I agree if we enable VLAN tag strip for the i40e driver, we can get around this problem
> but then all packets will be considered as received on the main interface.
> 
> I see...
> As far as I  know, when VLAN stripping is disabled,  i40e RXD doesn't contain information does that packet contain a VLAN or not.
> So, if enabling vlan stripping is not an option for you guys, then I don't see any other way on i40e to recognise is that  VLAN packet or
> not, except then parse the packet in SW.
> Helin, please correct me here, if I am missing something here.
> Thanks
> Konstantin
> 
> >
> > Regards,
> > John
> >
> > -----Original Message-----
> > From: Ananyev, Konstantin [mailto:konstantin.ananyev@intel.com]
> > Sent: Wednesday, May 25, 2016 10:35 AM
> > To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan
> > Cc: vpp-dev
> > Subject: RE: [vpp-dev] VLAN packets dropped... ?
> >
> >
> > >
> > > Since this is the XL710 40GE NIC, I suppose the DPDK driver involved would be the i40e driver and not ixgbe for 10GE NICs.
> >
> > Yes, I understand that you are facing problem on i40e, not ixgbe.
> > And the problem is that for i40e PKT_RX_VLAN_PKT flag is not set in mbuf->ol_flags, correct?
> > That's why I asked: are you running it with  rte_eth_conf.rxmode.hw_vlan_strip==0 or not?
> > If yes, you can try with rte_eth_conf.rxmode.hw_vlan_strip=1 and see would it help you.
> >
> > >
> > > As explained before, ixgbe driver had the inverse patch added. It
> > > does recognize VLAN tag in the packet and set PKT_RX_VLAN_PKT offload flag  properly:
> >
> > That patch has nothing to do with PKT_RX_VLAN_PKT and i40e.
> > So I don't think it is related to that problem at all.
> > Konstantin
> >
> > >
> > > 00:01:02:132370: dpdk-input
> > >   TenGigabitEthernet5/0/0 rx queue 0
> > >   buffer 0x44cff: current data 0, length 96, free-list 0, totlen-nifb 0, trace 0x0
> > >   PKT MBUF: port 3, nb_segs 1, pkt_len 96
> > >     buf_len 2176, data_len 96, ol_flags 0x1,
> > >     packet_type 0x210
> > >     Packet Offload Flags
> > >       PKT_RX_VLAN_PKT (0x0001) RX packet is a 802.1q VLAN packet
> > >     Packet Types
> > >       RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
> > >       RTE_PTYPE_L4_UDP (0x0200) UDP packet
> > >   IP4: 00:00:00:00:00:aa -> 90:e2:ba:6e:e7:dc vlan 1101
> > >   UDP: 28.0.0.1 -> 29.0.0.254
> > >     tos 0x00, ttl 64, length 78, checksum 0x40a1
> > >     fragment id 0x0000
> > >   UDP: 63 -> 63
> > >     length 58, checksum 0x5b76
> > > 00:01:02:132391: ethernet-input
> > >   IP4: 00:00:00:00:00:aa -> 90:e2:ba:6e:e7:dc vlan 1101
> > > 00:01:02:132408: error-drop
> > >   ethernet-input: unknown vlan
> > >
> > > You can see from above the packet was passed to ethernet-input node
> > > and dropped because sub-interface for VLAN 1101 is not set up.
> > >
> > > Regards,
> > > John
> > >
> > > -----Original Message-----
> > > From: Ananyev, Konstantin [mailto:konstantin.ananyev@intel.com]
> > > Sent: Wednesday, May 25, 2016 4:46 AM
> > > To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan
> > > Cc: vpp-dev
> > > Subject: RE: [vpp-dev] VLAN packets dropped... ?
> > >
> > >
> > > Hi,
> > > I don’t think ptype recognition on ixgbe vRX is somehow related to that.
> > > Are you guys running FVL with rte_eth_conf.rxmode.hw_vlan_strip==0?
> > > If so, can you set it to 1 and see would it help?
> > > Konstantin
> > >
> > > From: vpp-dev-bounces@lists.fd.io
> > > [mailto:vpp-dev-bounces@lists.fd.io]
> > > On Behalf Of John Lo (loj)
> > > Sent: Wednesday, May 25, 2016 1:56 AM
> > > To: Wiles, Keith; Chandrasekar Kannan
> > > Cc: vpp-dev
> > > Subject: Re: [vpp-dev] VLAN packets dropped... ?
> > >
> > > I did put in a reverse patch, provided by Damjan, for this in fd.io:
> > > https://gerrit.fd.io/r/#/c/941/1/dpdk/dpdk-16.04_patches/0013-Revert
> > > -i xgbe-fix-packet-type-from-vector-Rx.patch
> > >
> > > I know this fixed the 10GE NIC behavior. Perhaps the 40GE NIC is using a different DPDK driver?
> > >
> > > Regards,
> > > John
> > >
> > > From: Wiles, Keith [mailto:keith.wiles@intel.com]
> > > Sent: Tuesday, May 24, 2016 8:30 PM
> > > To: John Lo (loj); Chandrasekar Kannan
> > > Cc: vpp-dev
> > > Subject: Re: [vpp-dev] VLAN packets dropped... ?
> > >
> > >
> > > The commit ID is
> > >
> > > commit d9a2009a81089093645fea2e04b51dd37edf3e6f
> > > Author: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > > Date:   Tue Mar 22 14:30:17 2016 +0000
> > >
> > >     ixgbe: fix packet type from vector Rx
> > >
> > >     Current vector RX can't always set the packet_type properly.
> > >     To be more specific:
> > >     a) it never sets RTE_PTYPE_L2_ETHER
> > >     b) it doesn't handle tunnel ipv4/ipv6 case correctly.
> > >     c) it doesn't check is IXGBE_RXDADV_PKTTYPE_ETQF set or not.
> > >     While a) is pretty easy to fix, b) and c) are not that
> > > straightforward
> > >     in terms of SIMD ops (specially b).
> > >     So far I wasn't able to make vRX support packet_type properly
> > > without
> > >     noticeable performance loss.
> > >     So for now, just remove that functionality from vector RX and
> > >     update dev_supported_ptypes_get().
> > >
> > >     Fixes: 396254175854 ("mbuf: redefine packet type")
> > >
> > >     Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
> > >     Acked-by: Cunming Liang <cunming.liang@intel.com>
> > >
> > > It appears the change was only down to the vector version in the RX routine, could move to the none-vector version I guess.
> > >
> > > I would think adding a patch or adding the code to VPP DPDK init to use the RX callback to setup things correctly.
> > >
> > > Regards,
> > > Keith
> > >
> > >
> > > From: <vpp-dev-bounces@lists.fd.io> on behalf of Keith Wiles
> > > <keith.wiles@intel.com>
> > > Date: Tuesday, May 24, 2016 at 7:19 PM
> > > To: "John Lo (loj)" <loj@cisco.com>, Chandrasekar Kannan
> > > <ckannan@console.to>
> > > Cc: vpp-dev <vpp-dev@lists.fd.io>
> > > Subject: Re: [vpp-dev] VLAN packets dropped... ?
> > >
> > >
> > > Here is the starting thread:
> > > http://dpdk.org/ml/archives/dev/2016-April/037837.html
> > >
> > >
> > > Regards,
> > > Keith
> > >
> > >
> > > From: <vpp-dev-bounces@lists.fd.io> on behalf of Keith Wiles
> > > <keith.wiles@intel.com>
> > > Date: Tuesday, May 24, 2016 at 7:16 PM
> > > To: "John Lo (loj)" <loj@cisco.com>, Chandrasekar Kannan
> > > <ckannan@console.to>
> > > Cc: vpp-dev <vpp-dev@lists.fd.io>
> > > Subject: Re: [vpp-dev] VLAN packets dropped... ?
> > >
> > > The flags in the rte_mbuf were changed a before 16.04, which removed
> > > the ptype testing and possible setting this flag. I thought a patch to revert that change was done, did that not happen.
> > >
> > > The other solution is to add an option to DPDK to restore the
> > > previous behavior for this type of problem. The only other way is to
> > > have a installed in the PMD to setup the flags correctly for VPP.
> > > This would require adding a routine to VPP and setting the call back
> > > in the
> > driver.
> > >
> > > I really thought a patch was created, does anyone know?
> > >
> > > Regards,
> > > Keith
> > >
> > >
> > > From: "John Lo (loj)" <loj@cisco.com>
> > > Date: Tuesday, May 24, 2016 at 6:53 PM
> > > To: Chandrasekar Kannan <ckannan@console.to>
> > > Cc: Keith Wiles <keith.wiles@intel.com>, vpp-dev
> > > <vpp-dev@lists.fd.io>
> > > Subject: RE: [vpp-dev] VLAN packets dropped... ?
> > >
> > > I believe the problem is due to 40GE DPDK driver not parsing VLAN
> > > packet properly and setting the packet offload flag PKT_RX_VLAN_PKT,
> > > causing dpdk-input node to send packet directly to ip4-input instead
> > > of ethernet-input. If this flag is set, then the packet will be
> > > passed to ethernet-input which will then parse the VLAN tag in the
> > packet to perform sub-interface lookup and find the start of IP header
> > in the packet properly. Following is an example trace of a correctly marked packet:
> > >
> > > 23:11:09:855916: dpdk-input
> > >   TenGigE0/0/0/0 rx queue 0
> > >   buffer 0x23f6c0: current data 0, length 118, free-list 0,
> > > totlen-nifb 0, trace 0x0
> > >   PKT MBUF: port 0, nb_segs 1, pkt_len 118
> > >     buf_len 2176, data_len 118, ol_flags 0x1,
> > >     packet_type 0x10
> > >     Packet Offload Flags
> > >       PKT_RX_VLAN_PKT (0x0001) RX packet is a 802.1q VLAN packet
> > >     Packet Types
> > >       RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension
> > > headers
> > >   IP4: 74:a0:2f:e2:d5:14 -> 90:e2:ba:9f:8b:cc vlan 101
> > >   ICMP: 130.1.1.1 -> 130.1.1.2
> > >     tos 0x00, ttl 255, length 100, checksum 0xac4b
> > >     fragment id 0x0948
> > >   ICMP echo_request checksum 0x900d
> > >
> > > Hoping someone familiar with the 40GE DPDK driver can comment further.
> > >
> > > Regards,
> > > John
> > >
> > > From: Chandrasekar Kannan [mailto:ckannan@console.to]
> > > Sent: Tuesday, May 24, 2016 6:57 PM
> > > To: John Lo (loj)
> > > Cc: Wiles, Keith; vpp-dev
> > > Subject: Re: [vpp-dev] VLAN packets dropped... ?
> > >
> > > Yes. Subinterface has been configured like this  ...
> > >
> > > vppctl set interface ip address del FortyGigabitEthernet83/0/0 all
> > > vppctl create sub FortyGigabitEthernet83/0/0 900 vppctl set
> > > interface ip address FortyGigabitEthernet83/0/0.900 6.0.0.1/24
> > > vppctl set interface ip address FortyGigabitEthernet83/0/0.900
> > > 7.0.0.1/24 vppctl set interface state FortyGigabitEthernet83/0/0 up
> > > vppctl set interface state FortyGigabitEthernet83/0/0.900 up vppctl
> > > set ip arp static
> > > FortyGigabitEthernet83/0/0.900 7.0.0.2 3c:fd:fe:9c:c3:88
> > >
> > > --------------------------------------------------
> > > Packet 4
> > >
> > > 00:02:23:366879: dpdk-input
> > >   FortyGigabitEthernet83/0/0 rx queue 0
> > >   buffer 0x10e6b: current data 14, length 50, free-list 0,
> > > totlen-nifb 0, trace 0x3
> > >   PKT MBUF: port 0, nb_segs 1, pkt_len 64
> > >     buf_len 2176, data_len 64, ol_flags 0x0,
> > >     packet_type 0x291
> > >     Packet Types
> > >       RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
> > >       RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or
> > > without extension headers
> > >       RTE_PTYPE_L4_UDP (0x0200) UDP packet
> > >   IP4: 3c:fd:fe:9c:c3:88 -> 3c:fd:fe:9d:a0:20 vlan 900
> > >   UDP: 6.0.0.2 -> 7.0.0.2
> > >     tos 0x00, ttl 64, length 46, checksum 0x2dbc
> > >     fragment id 0x0000, flags DONT_FRAGMENT
> > >   UDP: 1024 -> 2001
> > >     length 26, checksum 0x0000
> > > 00:02:23:366894: ip4-input-no-checksum
> > >   IP6_HOP_BY_HOP_OPTIONS: 64.17.45.188 -> 6.0.0.2
> > >     version 0, header length 12
> > >     tos 0x84, ttl 0, length 2048, checksum 0x4000 (should be 0xaf4d)
> > >     fragment id 0x4500 offset 368, flags
> > > 00:02:23:366910: error-drop
> > >   ip4-input: ip4 length > l2 length
> > >
> > > --------------------------------------------------------------
> > >
> > >
> > >
> > > On Tue, May 24, 2016 at 3:12 PM, John Lo (loj) <loj@cisco.com> wrote:
> > > Was a sub-interface created with the expected VLAN tag 900? An
> > > example of creating a sub-interface and bring it up would be something like:
> > >
> > > create sub GigabitEthernet1/0/0 900
> > > set int state GigabitEthernet1/0/0 up set int state
> > > GigabitEthernet1/0/0.900 up
> > >
> > > Regards,
> > > John
> > >
> > >
> > > From: vpp-dev-bounces@lists.fd.io
> > > [mailto:vpp-dev-bounces@lists.fd.io]
> > > On Behalf Of Wiles, Keith
> > > Sent: Tuesday, May 24, 2016 5:59 PM
> > > To: Chandrasekar Kannan; vpp-dev
> > > Subject: Re: [vpp-dev] VLAN packets dropped... ?
> > >
> > > I am starting to think the unknown VLAN is not accounted for in the processing of the input frame in the VPP DPDK interface code.
> > > vnet/vnet/devices/dpdk/node.c
> > >
> > > The version and header length are wrong in the output below, which
> > > leads me to believe the starting IPv4 header location is wrong. I
> > > have not parsed all of the code in node.c:dpdk_device_input() yet,
> > > but I do not think the l3_offset is being adjusted to the correct
> > offset????
> > >
> > > Regards,
> > > Keith
> > >
> > >
> > > From: <vpp-dev-bounces@lists.fd.io> on behalf of Chandrasekar Kannan
> > > <ckannan@console.to>
> > > Date: Tuesday, May 24, 2016 at 3:40 PM
> > > To: vpp-dev <vpp-dev@lists.fd.io>
> > > Subject: [vpp-dev] VLAN packets dropped... ?
> > >
> > > Hi,
> > >
> > > I was trying to setup VLAN subinterfaces with VPP but it appears the
> > > packets are dropped at ingress.  I'm tried with both pktgen-dpdk and
> > > trex for traffic generation just to rule out any corruption with
> > pkt generation. But both are producing the same errors with vpp.
> > >
> > > yaml file from trex...
> > >
> > > ------------------------------
> > > [root@node04 scripts]# cat cap2/ipv4_vlan.yaml
> > > - duration : 10
> > >   generator :
> > >           distribution : "seq"
> > >           clients_start : "16.0.0.1"
> > >           clients_end   : "16.0.1.255"
> > >           servers_start : "48.0.0.1"
> > >           servers_end   : "48.0.0.255"
> > >           clients_per_gb : 201
> > >           min_clients    : 101
> > >           dual_port_mask : "1.0.0.0"
> > >           tcp_aging      : 1
> > >           udp_aging      : 1
> > >   mac        : [0x0,0x0,0x0,0x1,0x0,0x00]
> > >   vlan       : { enable : 1  ,  vlan0 : 900 , vlan1 : 1100 }
> > >   cap_info :
> > >      - name: cap2/udp_64B.pcap
> > >        cps   : 1000.0
> > >        ipg   : 10000
> > >        rtt   : 10000
> > >        w     : 4
> > >        limit : 20
> > > [root@node04 scripts]#
> > > ---------------------------------------------------------------
> > >
> > > vpp trace...
> > >
> > > 00:02:21:576810: dpdk-input
> > >   FortyGigabitEthernet83/0/0 rx queue 0
> > >   buffer 0x10e6b: current data 14, length 50, free-list 0,
> > > totlen-nifb 0, trace 0x1
> > >   PKT MBUF: port 0, nb_segs 1, pkt_len 64
> > >     buf_len 2176, data_len 64, ol_flags 0x0,
> > >     packet_type 0x291
> > >     Packet Types
> > >       RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
> > >       RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or
> > > without extension headers
> > >       RTE_PTYPE_L4_UDP (0x0200) UDP packet
> > >   IP4: 3c:fd:fe:9c:c3:88 -> 3c:fd:fe:9d:a0:20 vlan 900
> > >   UDP: 16.0.1.0 -> 48.0.0.128
> > >     tos 0x00, ttl 64, length 46, checksum 0xf93f
> > >     fragment id 0x0000, flags DONT_FRAGMENT
> > >   UDP: 1024 -> 2001
> > >     length 26, checksum 0x0000
> > > 00:02:21:576817: ip4-input-no-checksum
> > >   IP6_HOP_BY_HOP_OPTIONS: 64.17.249.63 -> 16.0.1.0
> > >     version 0, header length 12
> > >     tos 0x84, ttl 0, length 2048, checksum 0x4000 (should be 0xaf4d)
> > >     fragment id 0x4500 offset 368, flags
> > > 00:02:21:576828: error-drop
> > >   ip4-input: ip4 length > l2 length
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >