[dpdk-dev] Wrong TCP Checkum computed by hardware

[dpdk-dev] Wrong TCP Checkum computed by hardware

[Padam Jeet Singh]

Hi,

While implementing a sample packet generation code, I am setting up the offload flags so that the hardware (82599ES in my case) computes the IP and TCP Checksums. However, it seems that only the IP checksum is computed whereas the TCP Checksum is computed, but computed wrong. Here is a snippet of what I doing:

/* First reset all offload flags */
mbuf->ol_flags = (uint16_t) (mbuf->ol_flags & (~PKT_TX_OFFLOAD_MASK));
/* Then enable IP Checksum offload */
mbuf->ol_flags |= PKT_TX_IP_CKSUM;
/* This is required to be set to 0 */
ipv4hdr->hdr_checksum = 0;
mbuf->ol_flags |= PKT_TX_TCP_CKSUM;
/* Compute the PSD header */
tcphdr->cksum = rte_ipv4_phdr_cksum(ipv4hdr);

When I replace the checksum computation from hardware-offload to a software implementation, the checksum computed is correct. I have verified both the cases with Wireshark on the system receiving the packets.

Any hint what could I be doing wrong here?

Thanks,
Padam

Re: [dpdk-dev] Wrong TCP Checkum computed by hardware

[Matthew Hall]

On Wed, Oct 28, 2015 at 12:20:22PM +0530, Padam Jeet Singh wrote:
> Any hint what could I be doing wrong here?

When this kind of stuff doesn't work it often will depend on the exact version 
of card, chip, etc. if there are any errata.

So you might want to collect the specifics of the board with lspci -v, 
ethtool, and pulling it out to check the chip and board revisions.

In addition check over the example apps and see how things work there compared 
with your own code. Often the DPDK interfaces are kind of complex and small 
pointer or mbuf manipulation mistakes can cause very odd results.

Matthew.

Re: [dpdk-dev] Wrong TCP Checkum computed by hardware

[Padam Jeet Singh]

On 28-Oct-2015, at 12:27 pm, Matthew Hall <mhall@mhcomputing.net> wrote:
> 
> On Wed, Oct 28, 2015 at 12:20:22PM +0530, Padam Jeet Singh wrote:
>> Any hint what could I be doing wrong here?
> 
> When this kind of stuff doesn't work it often will depend on the exact version 
> of card, chip, etc. if there are any errata.
> 

82599ES or ixgbe PMD has not had any bug fixes related to offload - not at least what I can see in the git commits.

One important finding: Issue only comes when I do TX VLAN offload using PKT_TX_VLAN_PKT (fill vlan tci, l2_len, l3_len with the VLAN ID, sizeof(struct ether_hdr), sizeof(struct ipv4_hdr) respectively.

So basically VLAN OFFLOAD + IP CSUM OFFLOAD + TCP CSUM OFFLOAD causes the TCP checksum to be computed wrong. VLAN Offload + IP CSUM Offload + TCP CSUM in Software produces correct results. I am suspecting this to be ixgbe driver bug now as nothing in my code can trigger this behaviour.

> So you might want to collect the specifics of the board with lspci -v, 
> ethtool, and pulling it out to check the chip and board revisions.

02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
	Flags: bus master, fast devsel, latency 0, IRQ 32
	Memory at c7d20000 (64-bit, non-prefetchable) [size=128K]
	Memory at c7d44000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [e0] Vital Product Data
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-90-0b-ff-ff-3f-19-d0
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
	Kernel driver in use: igb_uio
	Kernel modules: ixgbe

> 
> In addition check over the example apps and see how things work there compared 
> with your own code. Often the DPDK interfaces are kind of complex and small 
> pointer or mbuf manipulation mistakes can cause very odd results.
> 

None of the sample code addresses the scenario which I have = VLAN offload + IP + TCP offload.

> Matthew.

Thanks,
Padam

Re: [dpdk-dev] Wrong TCP Checkum computed by hardware

[Liu, Jijiang]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Padam Jeet Singh
> Sent: Wednesday, October 28, 2015 3:20 PM
> To: Matthew Hall
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
> 

Did you set the txq_flags?
If the flag is not set, the default value is IXGBE_SIMPLE_FLAGS, and the any TX offloads do not work.

See the following codes in ixgbe_rxtx.c file

/* Use a simple Tx queue (no offloads, no multi segs) if possible */
	if (((txq->txq_flags & IXGBE_SIMPLE_FLAGS) == IXGBE_SIMPLE_FLAGS)
			&& (txq->tx_rs_thresh >= RTE_PMD_IXGBE_TX_MAX_BURST)) {
		PMD_INIT_LOG(INFO, "Using simple tx code path");
                      ...
		dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
	} else {
	             ...
		dev->tx_pkt_burst = ixgbe_xmit_pkts;
	}


--Jijiang Liu

Re: [dpdk-dev] Wrong TCP Checkum computed by hardware

[Padam Jeet Singh]

> On 28-Oct-2015, at 1:31 pm, Liu, Jijiang <jijiang.liu@intel.com> wrote:
> 
> 
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Padam Jeet Singh
>> Sent: Wednesday, October 28, 2015 3:20 PM
>> To: Matthew Hall
>> Cc: dev@dpdk.org
>> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
>> 
> 
> Did you set the txq_flags?
> If the flag is not set, the default value is IXGBE_SIMPLE_FLAGS, and the any TX offloads do not work.
> 
> See the following codes in ixgbe_rxtx.c file
> 
> /* Use a simple Tx queue (no offloads, no multi segs) if possible */
> 	if (((txq->txq_flags & IXGBE_SIMPLE_FLAGS) == IXGBE_SIMPLE_FLAGS)
> 			&& (txq->tx_rs_thresh >= RTE_PMD_IXGBE_TX_MAX_BURST)) {
> 		PMD_INIT_LOG(INFO, "Using simple tx code path");
>                      ...
> 		dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
> 	} else {
> 	             ...
> 		dev->tx_pkt_burst = ixgbe_xmit_pkts;
> 	}
> 
> 
> --Jijiang Liu

I initialise the queue with the following structure:

static const struct rte_eth_txconf tx_conf = {
	.tx_thresh = {
		.pthresh = 32,  /* Ring prefetch threshold */
		.hthresh = 0,   /* Ring host threshold */
		.wthresh = 0,   /* Ring writeback threshold */
	},
	.tx_free_thresh = 0,    /* Use PMD default values */
	.tx_rs_thresh = 0,      /* Use PMD default values */
};

This would set the txq_flags to zero - so the tx_pkt_burst function would always point to ixgbe_xmit_pkts. Also, as observed only TCP checksum is computed wrong when there is VLAN TX Offload + IP Offload + TCP offload bits set.  VLAN TX Offload + IP Offload + TCP CKSUM in software generates correct packet on the wire.

Thanks,
Padam

Re: [dpdk-dev] Wrong TCP Checkum computed by hardware

[Liu, Jijiang]

> -----Original Message-----
> From: Padam Jeet Singh [mailto:padam.singh@inventum.net]
> Sent: Wednesday, October 28, 2015 4:12 PM
> To: Liu, Jijiang
> Cc: dev@dpdk.org; Matthew Hall
> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
> 
> 
> > On 28-Oct-2015, at 1:31 pm, Liu, Jijiang <jijiang.liu@intel.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Padam Jeet Singh
> >> Sent: Wednesday, October 28, 2015 3:20 PM
> >> To: Matthew Hall
> >> Cc: dev@dpdk.org
> >> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
> >>
> >
> > Did you set the txq_flags?
> > If the flag is not set, the default value is IXGBE_SIMPLE_FLAGS, and the any
> TX offloads do not work.
> >
> > See the following codes in ixgbe_rxtx.c file
> >
> > /* Use a simple Tx queue (no offloads, no multi segs) if possible */
> > 	if (((txq->txq_flags & IXGBE_SIMPLE_FLAGS) == IXGBE_SIMPLE_FLAGS)
> > 			&& (txq->tx_rs_thresh >=
> RTE_PMD_IXGBE_TX_MAX_BURST)) {
> > 		PMD_INIT_LOG(INFO, "Using simple tx code path");
> >                      ...
> > 		dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
> > 	} else {
> > 	             ...
> > 		dev->tx_pkt_burst = ixgbe_xmit_pkts;
> > 	}
> >
> >
> > --Jijiang Liu
> 
> I initialise the queue with the following structure:
> 
> static const struct rte_eth_txconf tx_conf = {
> 	.tx_thresh = {
> 		.pthresh = 32,  /* Ring prefetch threshold */
> 		.hthresh = 0,   /* Ring host threshold */
> 		.wthresh = 0,   /* Ring writeback threshold */
> 	},
> 	.tx_free_thresh = 0,    /* Use PMD default values */
> 	.tx_rs_thresh = 0,      /* Use PMD default values */
> };
> 
> This would set the txq_flags to zero - so the tx_pkt_burst function would
> always point to ixgbe_xmit_pkts. Also, as observed only TCP checksum is
> computed wrong when there is VLAN TX Offload + IP Offload + TCP offload
> bits set.  VLAN TX Offload + IP Offload + TCP CKSUM in software generates
> correct packet on the wire.

I don't think the txq_flags is 0 if you just initialized the struct rte_eth_txconf like that.

> Thanks,
> Padam

Re: [dpdk-dev] Wrong TCP Checkum computed by hardware

[Padam Jeet Singh]

> On 28-Oct-2015, at 1:46 pm, Liu, Jijiang <jijiang.liu@intel.com> wrote:
> 
> 
> 
>> -----Original Message-----
>> From: Padam Jeet Singh [mailto:padam.singh@inventum.net]
>> Sent: Wednesday, October 28, 2015 4:12 PM
>> To: Liu, Jijiang
>> Cc: dev@dpdk.org; Matthew Hall
>> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
>> 
>> 
>>> On 28-Oct-2015, at 1:31 pm, Liu, Jijiang <jijiang.liu@intel.com> wrote:
>>> 
>>> 
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Padam Jeet Singh
>>>> Sent: Wednesday, October 28, 2015 3:20 PM
>>>> To: Matthew Hall
>>>> Cc: dev@dpdk.org
>>>> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
>>>> 
>>> 
>>> Did you set the txq_flags?
>>> If the flag is not set, the default value is IXGBE_SIMPLE_FLAGS, and the any
>> TX offloads do not work.
>>> 
>>> See the following codes in ixgbe_rxtx.c file
>>> 
>>> /* Use a simple Tx queue (no offloads, no multi segs) if possible */
>>> 	if (((txq->txq_flags & IXGBE_SIMPLE_FLAGS) == IXGBE_SIMPLE_FLAGS)
>>> 			&& (txq->tx_rs_thresh >=
>> RTE_PMD_IXGBE_TX_MAX_BURST)) {
>>> 		PMD_INIT_LOG(INFO, "Using simple tx code path");
>>>                     ...
>>> 		dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
>>> 	} else {
>>> 	             ...
>>> 		dev->tx_pkt_burst = ixgbe_xmit_pkts;
>>> 	}
>>> 
>>> 
>>> --Jijiang Liu
>> 
>> I initialise the queue with the following structure:
>> 
>> static const struct rte_eth_txconf tx_conf = {
>> 	.tx_thresh = {
>> 		.pthresh = 32,  /* Ring prefetch threshold */
>> 		.hthresh = 0,   /* Ring host threshold */
>> 		.wthresh = 0,   /* Ring writeback threshold */
>> 	},
>> 	.tx_free_thresh = 0,    /* Use PMD default values */
>> 	.tx_rs_thresh = 0,      /* Use PMD default values */
>> };
>> 
>> This would set the txq_flags to zero - so the tx_pkt_burst function would
>> always point to ixgbe_xmit_pkts. Also, as observed only TCP checksum is
>> computed wrong when there is VLAN TX Offload + IP Offload + TCP offload
>> bits set.  VLAN TX Offload + IP Offload + TCP CKSUM in software generates
>> correct packet on the wire.
> 
> I don't think the txq_flags is 0 if you just initialized the struct rte_eth_txconf like that.

It’s declared as a global static, so it indeed is 0. I also added some debug around the init of the queue:

for (i = 0; i < tx; ++i) {
	    ret = rte_eth_tx_queue_setup(port, i, NB_TXD,
		rte_eth_dev_socket_id(port), &tx_conf);
	    RTE_LOG(INFO, APP, "Port %u TXQ[%d] txflags = %d\n", (unsigned)port,
	    	i, tx_conf.txq_flags);
	    if (ret < 0)
		rte_exit(EXIT_FAILURE, "Could not setup up TX queue %d for "
				"port%u (%d)", i, (unsigned)port, ret);
	}

And got the following result:

Oct 28 13:55:34 localhost fpnas[1322]: APP: Port 0 TXQ[0] txflags = 0
Oct 28 13:55:34 localhost fpnas[1322]: APP: Port 0 TXQ[1] txflags = 0
Oct 28 13:55:34 localhost fpnas[1322]: APP: Port 1 TXQ[0] txflags = 0
Oct 28 13:55:34 localhost fpnas[1322]: APP: Port 1 TXQ[1] txflags = 0
Oct 28 13:55:35 localhost fpnas[1322]: APP: Port 2 TXQ[0] txflags = 0
Oct 28 13:55:35 localhost fpnas[1322]: APP: Port 3 TXQ[0] txflags = 0


> 
>> Thanks,
>> Padam

Re: [dpdk-dev] Wrong TCP Checkum computed by hardware

[Liu, Jijiang]

> -----Original Message-----
> From: Padam Jeet Singh [mailto:padam.singh@inventum.net]
> Sent: Wednesday, October 28, 2015 4:31 PM
> To: Liu, Jijiang
> Cc: dev@dpdk.org; Matthew Hall
> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
> 
> 
> > On 28-Oct-2015, at 1:46 pm, Liu, Jijiang <jijiang.liu@intel.com> wrote:
> >
> >
> >
> >> -----Original Message-----
> >> From: Padam Jeet Singh [mailto:padam.singh@inventum.net]
> >> Sent: Wednesday, October 28, 2015 4:12 PM
> >> To: Liu, Jijiang
> >> Cc: dev@dpdk.org; Matthew Hall
> >> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
> >>
> >>
> >>> On 28-Oct-2015, at 1:31 pm, Liu, Jijiang <jijiang.liu@intel.com> wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Padam Jeet
> >>>> Singh
> >>>> Sent: Wednesday, October 28, 2015 3:20 PM
> >>>> To: Matthew Hall
> >>>> Cc: dev@dpdk.org
> >>>> Subject: Re: [dpdk-dev] Wrong TCP Checkum computed by hardware
> >>>>
> >>>
> >>> Did you set the txq_flags?
> >>> If the flag is not set, the default value is IXGBE_SIMPLE_FLAGS, and
> >>> the any
> >> TX offloads do not work.
> >>>
> >>> See the following codes in ixgbe_rxtx.c file
> >>>
> >>> /* Use a simple Tx queue (no offloads, no multi segs) if possible */
> >>> 	if (((txq->txq_flags & IXGBE_SIMPLE_FLAGS) == IXGBE_SIMPLE_FLAGS)
> >>> 			&& (txq->tx_rs_thresh >=
> >> RTE_PMD_IXGBE_TX_MAX_BURST)) {
> >>> 		PMD_INIT_LOG(INFO, "Using simple tx code path");
> >>>                     ...
> >>> 		dev->tx_pkt_burst = ixgbe_xmit_pkts_simple;
> >>> 	} else {
> >>> 	             ...
> >>> 		dev->tx_pkt_burst = ixgbe_xmit_pkts;
> >>> 	}
> >>>
> >>>
> >>> --Jijiang Liu
> >>
> >> I initialise the queue with the following structure:
> >>
> >> static const struct rte_eth_txconf tx_conf = {
> >> 	.tx_thresh = {
> >> 		.pthresh = 32,  /* Ring prefetch threshold */
> >> 		.hthresh = 0,   /* Ring host threshold */
> >> 		.wthresh = 0,   /* Ring writeback threshold */
> >> 	},
> >> 	.tx_free_thresh = 0,    /* Use PMD default values */
> >> 	.tx_rs_thresh = 0,      /* Use PMD default values */
> >> };
> >>
> >> This would set the txq_flags to zero - so the tx_pkt_burst function
> >> would always point to ixgbe_xmit_pkts. Also, as observed only TCP
> >> checksum is computed wrong when there is VLAN TX Offload + IP Offload
> >> + TCP offload bits set.  VLAN TX Offload + IP Offload + TCP CKSUM in
> >> software generates correct packet on the wire.
> >
> > I don't think the txq_flags is 0 if you just initialized the struct
> rte_eth_txconf like that.
> 
> It’s declared as a global static, so it indeed is 0. I also added some debug
> around the init of the queue:
> 
> for (i = 0; i < tx; ++i) {
> 	    ret = rte_eth_tx_queue_setup(port, i, NB_TXD,
> 		rte_eth_dev_socket_id(port), &tx_conf);
> 	    RTE_LOG(INFO, APP, "Port %u TXQ[%d] txflags = %d\n",
> (unsigned)port,
> 	    	i, tx_conf.txq_flags);
> 	    if (ret < 0)
> 		rte_exit(EXIT_FAILURE, "Could not setup up TX queue %d for
> "
> 				"port%u (%d)", i, (unsigned)port, ret);
> 	}
> 
> And got the following result:
> 
> Oct 28 13:55:34 localhost fpnas[1322]: APP: Port 0 TXQ[0] txflags = 0 Oct 28
> 13:55:34 localhost fpnas[1322]: APP: Port 0 TXQ[1] txflags = 0 Oct 28 13:55:34
> localhost fpnas[1322]: APP: Port 1 TXQ[0] txflags = 0 Oct 28 13:55:34 localhost
> fpnas[1322]: APP: Port 1 TXQ[1] txflags = 0 Oct 28 13:55:35 localhost
> fpnas[1322]: APP: Port 2 TXQ[0] txflags = 0 Oct 28 13:55:35 localhost
> fpnas[1322]: APP: Port 3 TXQ[0] txflags = 0
> 
Yes, you are correct, I just noticed you declare  it is static, If possible, send more codes to me, I can help you with this.
> >
> >> Thanks,
> >> Padam

Re: [dpdk-dev] Wrong TCP Checkum computed by hardware

[Padam Jeet Singh]

>> 
> Yes, you are correct, I just noticed you declare  it is static, If possible, send more codes to me, I can help you with this.
>>> 
>>>> Thanks,
>>>> Padam
> 


I have the following code:


mbuf->ol_flags = (uint16_t) (mbuf->ol_flags &
			(~PKT_TX_OFFLOAD_MASK));
mbuf->ol_flags |= PKT_TX_IP_CKSUM;
ipv4hdr->hdr_checksum = 0;
tcphdr = (struct tcp_hdr *)(pkt + sizeof(struct ether_hdr) +
						sizeof(struct ipv4_hdr));
#ifdef L4CSUM_SW

tcphdr->cksum = 0;
tcphdr->cksum = get_ipv4_udptcp_checksum(ipv4hdr,	(uint16_t*)tcphdr);

#else

mbuf->ol_flags |= PKT_TX_TCP_CKSUM;
if(!(mbuf->ol_flags & PKT_TX_IPV4))
	mbuf->ol_flags |= PKT_TX_IPV4;
tcphdr->cksum = 0;
tcphdr->cksum = rte_ipv4_phdr_cksum(ipv4hdr);

#endif

mbuf->pkt.vlan_macip.f.vlan_tci = 100;
mbuf->ol_flags |= PKT_TX_VLAN_PKT;

I have added a macro called L4CSUM_SW, if enabled does L4 Checksum computation in software. When I enable this, the packet received has the correct TCP checksum. It is important to note that the checksum is being computed, but is being computed wrong when TX_VLAN, IP_CKSUM and TCP_CKSUM are enabled.

When I do the same test with VLAN removed from the setup, the TCP checksum computation is correct. 

The system exhibiting this issue is a  Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz X 2, with 2 X 82599ES on a PCIe3 bus:
02:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
	Flags: bus master, fast devsel, latency 0, IRQ 32
	Memory at c7d20000 (64-bit, non-prefetchable) [size=128K]
	Memory at c7d44000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [e0] Vital Product Data
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 00-90-0b-ff-ff-3f-19-d0
	Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [160] Single Root I/O Virtualization (SR-IOV)
	Kernel driver in use: igb_uio
	Kernel modules: ixgbe