[dpdk-users] DPDK RX TCP checksum failed

[dpdk-users] DPDK RX TCP checksum failed

[jiangheng (H)]

[-- Attachment #1: Type: text/plain, Size: 1142 bytes --]

Hi all,

I tried using the checksum offloads feature in DPDK and it did not see working under virtual machine.

Port only support TCP checksum and do not support IP checksum:
rx_offload_capa = DEV_RX_OFFLOAD_TCP_CKSUM
tx_offload_capa = DEV_TX_OFFLOAD_TCP_CKSUM

so I config rxmode.offload txmode.offloads as below:
rxmode.offloads = DEV_RX_OFFLOAD_TCP_CKSUM
txmode.offloads = DEV_TX_OFFLOAD_TCP_CKSUM

For TX, I set the following parameters, it works good.
mbuf->l2_len = sizeof(*ethhdr)
mbuf->l3_len = ip header len
mbuf-ol_flags = RTE_MBUF_F_TX_IPV4 | RTE_MBUF_F_TX_TCP_CKSUM


For RX, It will execute the following code:
In drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function :
    if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
        hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
        if (hdr->csum_start <= hdrlen && l4_supported) {
            m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
        } else {

m->ol_flags set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing the TCP RX checksum failed.
How do I avoid the above code going into this branch?

Thanks,
Jinag.








[-- Attachment #2: Type: text/html, Size: 5709 bytes --]

Re: [dpdk-users] DPDK RX TCP checksum failed

[Stephen Hemminger]

On Sat, 26 Mar 2022 08:13:21 +0000
"jiangheng (H)" <jiangheng12@huawei.com> wrote:

> Hi all,
> 
> I tried using the checksum offloads feature in DPDK and it did not see working under virtual machine.
> 
> Port only support TCP checksum and do not support IP checksum:
> rx_offload_capa = DEV_RX_OFFLOAD_TCP_CKSUM
> tx_offload_capa = DEV_TX_OFFLOAD_TCP_CKSUM
> 
> so I config rxmode.offload txmode.offloads as below:
> rxmode.offloads = DEV_RX_OFFLOAD_TCP_CKSUM
> txmode.offloads = DEV_TX_OFFLOAD_TCP_CKSUM
> 
> For TX, I set the following parameters, it works good.
> mbuf->l2_len = sizeof(*ethhdr)
> mbuf->l3_len = ip header len
> mbuf-ol_flags = RTE_MBUF_F_TX_IPV4 | RTE_MBUF_F_TX_TCP_CKSUM


Virtio does not support IP checksum offload.  Because Virtio passes
packets to Linux kernel, and Linux kernel does not do IP checksum offload.
The IP checksum is so trivial it is faster for most things to just
do it in software; the header is only 20 bytes and it will be in cache.

You should always check device capability before enabling an offload.


> For RX, It will execute the following code:
> In drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function :
>     if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
>         hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
>         if (hdr->csum_start <= hdrlen && l4_supported) {
>             m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
>         } else {
> 
> m->ol_flags set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing the TCP RX checksum failed.
> How do I avoid the above code going into this branch?
> 

If you want TCP checksum offload you have to set RTE_ETH_RX_OFFLOAD_TCP_CKSUM
in the rxmode when port is configured.  This will tell virtio to ask the
host to do rx offload. Again, virtio does not do IP checksum offload and
you should always query device capability first.

答复: [dpdk-users] DPDK RX TCP checksum failed

[jiangheng (H)]

> On Sat, 26 Mar 2022 08:13:21 +0000
> "jiangheng (H)" <jiangheng12@huawei.com> wrote:
> 
> > Hi all,
> >
> > I tried using the checksum offloads feature in DPDK and it did not see
> working under virtual machine.
> >
> > Port only support TCP checksum and do not support IP checksum:
> > rx_offload_capa = DEV_RX_OFFLOAD_TCP_CKSUM tx_offload_capa =
> > DEV_TX_OFFLOAD_TCP_CKSUM
> >
> > so I config rxmode.offload txmode.offloads as below:
> > rxmode.offloads = DEV_RX_OFFLOAD_TCP_CKSUM txmode.offloads =
> > DEV_TX_OFFLOAD_TCP_CKSUM
> >
> > For TX, I set the following parameters, it works good.
> > mbuf->l2_len = sizeof(*ethhdr)
> > mbuf->l3_len = ip header len
> > mbuf-ol_flags = RTE_MBUF_F_TX_IPV4 | RTE_MBUF_F_TX_TCP_CKSUM
> 
> 
> Virtio does not support IP checksum offload.  Because Virtio passes
> packets to Linux kernel, and Linux kernel does not do IP checksum offload.
> The IP checksum is so trivial it is faster for most things to just do it in
> software; the header is only 20 bytes and it will be in cache.
> 
> You should always check device capability before enabling an offload.
> 
> 
> > For RX, It will execute the following code:
> > In drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function :
> >     if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
> >         hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
> >         if (hdr->csum_start <= hdrlen && l4_supported) {
> >             m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
> >         } else {
> >
> > m->ol_flags set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing the TCP
> RX checksum failed.
> > How do I avoid the above code going into this branch?
> >
> 
> If you want TCP checksum offload you have to set
> RTE_ETH_RX_OFFLOAD_TCP_CKSUM in the rxmode when port is
> configured.  This will tell virtio to ask the host to do rx offload. Again,
> virtio does not do IP checksum offload and you should always query
> device capability first.

I have queried device capability and it tell me it supports tcp checksum, not supports ip checksum. 
So I have set RTE_ETH_RX_OFFLOAD_TCP_CKSUM(DEV_RX_OFFLOAD_TCP_CKSUM) flag when port is configured(use rte_eth_dev_configure function)
But RX checksum still failed in below branch:
drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function:
if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
	hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
	if (hdr->csum_start <= hdrlen && l4_supported) {
		if (hdr->csum_start <= hdrlen && l4_supported) {
			m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
		} else {
Hdr->csum_start <= hdrlen, m->ol_flags will set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing RX checksum failed.

Re: [dpdk-users] DPDK RX TCP checksum failed

[Harold Huang]

On Mon, Mar 28, 2022 at 9:50 AM jiangheng (H) <jiangheng12@huawei.com> wrote:
>
> > On Sat, 26 Mar 2022 08:13:21 +0000
> > "jiangheng (H)" <jiangheng12@huawei.com> wrote:
> >
> > > Hi all,
> > >
> > > I tried using the checksum offloads feature in DPDK and it did not see
> > working under virtual machine.
> > >
> > > Port only support TCP checksum and do not support IP checksum:
> > > rx_offload_capa = DEV_RX_OFFLOAD_TCP_CKSUM tx_offload_capa =
> > > DEV_TX_OFFLOAD_TCP_CKSUM
> > >
> > > so I config rxmode.offload txmode.offloads as below:
> > > rxmode.offloads = DEV_RX_OFFLOAD_TCP_CKSUM txmode.offloads =
> > > DEV_TX_OFFLOAD_TCP_CKSUM
> > >
> > > For TX, I set the following parameters, it works good.
> > > mbuf->l2_len = sizeof(*ethhdr)
> > > mbuf->l3_len = ip header len
> > > mbuf-ol_flags = RTE_MBUF_F_TX_IPV4 | RTE_MBUF_F_TX_TCP_CKSUM
> >
> >
> > Virtio does not support IP checksum offload.  Because Virtio passes
> > packets to Linux kernel, and Linux kernel does not do IP checksum offload.
> > The IP checksum is so trivial it is faster for most things to just do it in
> > software; the header is only 20 bytes and it will be in cache.
> >
> > You should always check device capability before enabling an offload.
> >
> >
> > > For RX, It will execute the following code:
> > > In drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function :
> > >     if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
> > >         hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
> > >         if (hdr->csum_start <= hdrlen && l4_supported) {
> > >             m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
> > >         } else {
> > >
> > > m->ol_flags set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing the TCP
> > RX checksum failed.
> > > How do I avoid the above code going into this branch?
> > >
> >
> > If you want TCP checksum offload you have to set
> > RTE_ETH_RX_OFFLOAD_TCP_CKSUM in the rxmode when port is
> > configured.  This will tell virtio to ask the host to do rx offload. Again,
> > virtio does not do IP checksum offload and you should always query
> > device capability first.
>
> I have queried device capability and it tell me it supports tcp checksum, not supports ip checksum.
> So I have set RTE_ETH_RX_OFFLOAD_TCP_CKSUM(DEV_RX_OFFLOAD_TCP_CKSUM) flag when port is configured(use rte_eth_dev_configure function)
> But RX checksum still failed in below branch:
> drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function:
> if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
>         hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
>         if (hdr->csum_start <= hdrlen && l4_supported) {
>                 if (hdr->csum_start <= hdrlen && l4_supported) {
>                         m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
>                 } else {
> Hdr->csum_start <= hdrlen, m->ol_flags will set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing RX checksum failed.

Could you show the hdr->csum_start and hdrlen value in your test case?
IIRC, hdr->csum_start should be hdr_lens.l2_len + hdr_lens.l3_len
according to virtio sepc. The virtio spec has an example as follows:

```
For example, consider a partially checksummed TCP (IPv4) packet. It
will have a 14 byte ether- net header and 20 byte IP header followed
by the TCP header (with the TCP checksum field 16 bytes into that
header). csum_start will be 14+20 = 34 (the TCP checksum includes the
header), and csum_offset will be 16.
```

-- 
Thanks, Harold.

答复: [dpdk-users] DPDK RX TCP checksum failed

[jiangheng (H)]

> > > On Sat, 26 Mar 2022 08:13:21 +0000
> > > "jiangheng (H)" <jiangheng12@huawei.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I tried using the checksum offloads feature in DPDK and it did not
> > > > see
> > > working under virtual machine.
> > > >
> > > > Port only support TCP checksum and do not support IP checksum:
> > > > rx_offload_capa = DEV_RX_OFFLOAD_TCP_CKSUM tx_offload_capa =
> > > > DEV_TX_OFFLOAD_TCP_CKSUM
> > > >
> > > > so I config rxmode.offload txmode.offloads as below:
> > > > rxmode.offloads = DEV_RX_OFFLOAD_TCP_CKSUM txmode.offloads =
> > > > DEV_TX_OFFLOAD_TCP_CKSUM
> > > >
> > > > For TX, I set the following parameters, it works good.
> > > > mbuf->l2_len = sizeof(*ethhdr)
> > > > mbuf->l3_len = ip header len
> > > > mbuf-ol_flags = RTE_MBUF_F_TX_IPV4 | RTE_MBUF_F_TX_TCP_CKSUM
> > >
> > >
> > > Virtio does not support IP checksum offload.  Because Virtio passes
> > > packets to Linux kernel, and Linux kernel does not do IP checksum offload.
> > > The IP checksum is so trivial it is faster for most things to just
> > > do it in software; the header is only 20 bytes and it will be in cache.
> > >
> > > You should always check device capability before enabling an offload.
> > >
> > >
> > > > For RX, It will execute the following code:
> > > > In drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function :
> > > >     if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
> > > >         hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
> > > >         if (hdr->csum_start <= hdrlen && l4_supported) {
> > > >             m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
> > > >         } else {
> > > >
> > > > m->ol_flags set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing the TCP
> > > RX checksum failed.
> > > > How do I avoid the above code going into this branch?
> > > >
> > >
> > > If you want TCP checksum offload you have to set
> > > RTE_ETH_RX_OFFLOAD_TCP_CKSUM in the rxmode when port is configured.
> > > This will tell virtio to ask the host to do rx offload. Again,
> > > virtio does not do IP checksum offload and you should always query
> > > device capability first.
> >
> > I have queried device capability and it tell me it supports tcp checksum, not
> supports ip checksum.
> > So I have set
> RTE_ETH_RX_OFFLOAD_TCP_CKSUM(DEV_RX_OFFLOAD_TCP_CKSUM)
> > flag when port is configured(use rte_eth_dev_configure function) But RX
> checksum still failed in below branch:
> > drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function:
> > if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
> >         hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
> >         if (hdr->csum_start <= hdrlen && l4_supported) {
> >                 if (hdr->csum_start <= hdrlen && l4_supported) {
> >                         m->ol_flags |=
> RTE_MBUF_F_RX_L4_CKSUM_NONE;
> >                 } else {
> > Hdr->csum_start <= hdrlen, m->ol_flags will set to
> RTE_MBUF_F_RX_L4_CKSUM_NONE, causing RX checksum failed.
> 
> Could you show the hdr->csum_start and hdrlen value in your test case?
> IIRC, hdr->csum_start should be hdr_lens.l2_len + hdr_lens.l3_len according to
> virtio sepc. The virtio spec has an example as follows:
> 
> ```
> For example, consider a partially checksummed TCP (IPv4) packet. It will have a
> 14 byte ether- net header and 20 byte IP header followed by the TCP header
> (with the TCP checksum field 16 bytes into that header). csum_start will be 14+20
> = 34 (the TCP checksum includes the header), and csum_offset will be 16.
> ```
> 
> --
> Thanks, Harold.

Yes, hdr->csum_start is 34, hdrlen is 54, so m->flags is RTE_MBUF_F_RX_L4_CKSUM_NONE. This cause rx checksum failed. But this line seems to be correct.
I suspect that hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM should not be 1, but I don’t know where this flag is set.

Re: [dpdk-users] DPDK RX TCP checksum failed

[fwefew 4t4tg]

[-- Attachment #1: Type: text/plain, Size: 2570 bytes --]

https://github.com/rodgarrison/reinvent
demonstrates ether/UDP chec-kum offload running on AWS which uses virtio. I
don't think TCP offload
is substantially different. Look here:

https://github.com/rodgarrison/reinvent/blob/main/integration_tests/reinvent_dpdk_udp/reinvent_dpdk_udp_integration_test.cpp#L345

for where the mbuf needs to be setup.

But I kind of agree with Stephan, although I haven't tested it, avoiding
more sets into mbuf which might involve cache misses it might be faster to
do by hand since the data is already in cache.

On Sat, Mar 26, 2022 at 2:59 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Sat, 26 Mar 2022 08:13:21 +0000
> "jiangheng (H)" <jiangheng12@huawei.com> wrote:
>
> > Hi all,
> >
> > I tried using the checksum offloads feature in DPDK and it did not see
> working under virtual machine.
> >
> > Port only support TCP checksum and do not support IP checksum:
> > rx_offload_capa = DEV_RX_OFFLOAD_TCP_CKSUM
> > tx_offload_capa = DEV_TX_OFFLOAD_TCP_CKSUM
> >
> > so I config rxmode.offload txmode.offloads as below:
> > rxmode.offloads = DEV_RX_OFFLOAD_TCP_CKSUM
> > txmode.offloads = DEV_TX_OFFLOAD_TCP_CKSUM
> >
> > For TX, I set the following parameters, it works good.
> > mbuf->l2_len = sizeof(*ethhdr)
> > mbuf->l3_len = ip header len
> > mbuf-ol_flags = RTE_MBUF_F_TX_IPV4 | RTE_MBUF_F_TX_TCP_CKSUM
>
>
> Virtio does not support IP checksum offload.  Because Virtio passes
> packets to Linux kernel, and Linux kernel does not do IP checksum offload.
> The IP checksum is so trivial it is faster for most things to just
> do it in software; the header is only 20 bytes and it will be in cache.
>
> You should always check device capability before enabling an offload.
>
>
> > For RX, It will execute the following code:
> > In drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function :
> >     if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
> >         hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
> >         if (hdr->csum_start <= hdrlen && l4_supported) {
> >             m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
> >         } else {
> >
> > m->ol_flags set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing the TCP RX
> checksum failed.
> > How do I avoid the above code going into this branch?
> >
>
> If you want TCP checksum offload you have to set
> RTE_ETH_RX_OFFLOAD_TCP_CKSUM
> in the rxmode when port is configured.  This will tell virtio to ask the
> host to do rx offload. Again, virtio does not do IP checksum offload and
> you should always query device capability first.
>
>

[-- Attachment #2: Type: text/html, Size: 3456 bytes --]

答复: [dpdk-users] DPDK RX TCP checksum failed

[jiangheng (H)]

[-- Attachment #1: Type: text/plain, Size: 3071 bytes --]

https://github.com/rodgarrison/reinvent/blob/main/integration_tests/reinvent_dpdk_udp/reinvent_dpdk_udp_integration_test.cpp#L345
this is for tx checksum, Tx checksum works good, Rx checksum failed.

发件人: fwefew 4t4tg [mailto:7532yahoo@gmail.com]
发送时间: 2022年3月28日 11:12
收件人: Stephen Hemminger <stephen@networkplumber.org>
抄送: jiangheng (H) <jiangheng12@huawei.com>; users@dpdk.org
主题: Re: [dpdk-users] DPDK RX TCP checksum failed

https://github.com/rodgarrison/reinvent
demonstrates ether/UDP chec-kum offload running on AWS which uses virtio. I don't think TCP offload
is substantially different. Look here:

https://github.com/rodgarrison/reinvent/blob/main/integration_tests/reinvent_dpdk_udp/reinvent_dpdk_udp_integration_test.cpp#L345

for where the mbuf needs to be setup.

But I kind of agree with Stephan, although I haven't tested it, avoiding more sets into mbuf which might involve cache misses it might be faster to do by hand since the data is already in cache.

On Sat, Mar 26, 2022 at 2:59 PM Stephen Hemminger <stephen@networkplumber.org<mailto:stephen@networkplumber.org>> wrote:
On Sat, 26 Mar 2022 08:13:21 +0000
"jiangheng (H)" <jiangheng12@huawei.com<mailto:jiangheng12@huawei.com>> wrote:

> Hi all,
>
> I tried using the checksum offloads feature in DPDK and it did not see working under virtual machine.
>
> Port only support TCP checksum and do not support IP checksum:
> rx_offload_capa = DEV_RX_OFFLOAD_TCP_CKSUM
> tx_offload_capa = DEV_TX_OFFLOAD_TCP_CKSUM
>
> so I config rxmode.offload txmode.offloads as below:
> rxmode.offloads = DEV_RX_OFFLOAD_TCP_CKSUM
> txmode.offloads = DEV_TX_OFFLOAD_TCP_CKSUM
>
> For TX, I set the following parameters, it works good.
> mbuf->l2_len = sizeof(*ethhdr)
> mbuf->l3_len = ip header len
> mbuf-ol_flags = RTE_MBUF_F_TX_IPV4 | RTE_MBUF_F_TX_TCP_CKSUM


Virtio does not support IP checksum offload.  Because Virtio passes
packets to Linux kernel, and Linux kernel does not do IP checksum offload.
The IP checksum is so trivial it is faster for most things to just
do it in software; the header is only 20 bytes and it will be in cache.

You should always check device capability before enabling an offload.


> For RX, It will execute the following code:
> In drivers/net/virtio/virtio_rxtx.c  virtio_rx_offload function :
>     if (hdr->flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
>         hdrlen = hdr_lens.l2_len + hdr_lens.l3_len + hdr_lens.l4_len;
>         if (hdr->csum_start <= hdrlen && l4_supported) {
>             m->ol_flags |= RTE_MBUF_F_RX_L4_CKSUM_NONE;
>         } else {
>
> m->ol_flags set to RTE_MBUF_F_RX_L4_CKSUM_NONE, causing the TCP RX checksum failed.
> How do I avoid the above code going into this branch?
>

If you want TCP checksum offload you have to set RTE_ETH_RX_OFFLOAD_TCP_CKSUM
in the rxmode when port is configured.  This will tell virtio to ask the
host to do rx offload. Again, virtio does not do IP checksum offload and
you should always query device capability first.

[-- Attachment #2: Type: text/html, Size: 7887 bytes --]