[dpdk-users] .hw_strip_crc weird behavior with small UDP packets

[dpdk-users] .hw_strip_crc weird behavior with small UDP packets

[Julien Castets]

Hi,


I dump the packets received with rte_pktmbuf_dump(stderr, pkt,
pkt->data_len) and I have a strange behavior related to CRC stripping
with small UDP packets. By "small", I mean below the minimal ethernet
payload size, ie. 46 bytes.

* case 1: when my device is configured with hw_strip_crc=1 and the UDP
data is "aaa\n" (4 bytes), my function outputs:

    dump mbuf at 0x0x7fe93bdc47c0, phys=2dbc4840, buf_len=1920
      pkt_len=60, ol_flags=2, nb_segs=1, in_port=1
      segment at 0x0x7fe93bdc47c0, data=0x0x7fe93bdc48c0, data_len=60
      Dump data at [0x7fe93bdc48c0], len=60
    00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E.
    00000010: 00 20 26 D5 00 00 39 11 3B FD D5 24 07 1F A3 AC | . &...9.;..$....
    00000020: 9F 0B EA 5D 10 92 00 0C 23 7F 61 61 61 0A 00 00 | ...]....#.aaa...
    00000030: 00 00 00 00 00 00 00 00 C6 63 C2 CF |  |  |  |  | .........c..

* case 2: when hw_strip_crc=1 and UDP data is
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
(66 bytes, ie. > to 46):

    dump mbuf at 0x0x7fe93bdd4fc0, phys=2dbd5040, buf_len=1920
      pkt_len=108, ol_flags=2, nb_segs=1, in_port=1
      segment at 0x0x7fe93bdd4fc0, data=0x0x7fe93bdd50c0, data_len=108
      Dump data at [0x7fe93bdd50c0], len=108
    00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E.
    00000010: 00 5E F3 BA 00 00 39 11 6E D9 D5 24 07 1F A3 AC | .^....9.n..$....
    00000020: 9F 0B EA 5D 10 92 00 4A 58 38 61 61 61 61 61 61 | ...]...JX8aaaaaa
    00000030: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000040: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000050: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000060: 61 61 61 61 61 61 61 61 61 61 61 0A |  |  |  |  | aaaaaaaaaaa.

* case 3: when hw_strip_crc=0 and data is "aaa\n" (4 bytes):

    dump mbuf at 0x0x7effcc996140, phys=2db961c0, buf_len=1920
      pkt_len=60, ol_flags=2, nb_segs=1, in_port=1
      segment at 0x0x7effcc996140, data=0x0x7effcc996240, data_len=60
      Dump data at [0x7effcc996240], len=60
    00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E.
    00000010: 00 20 A9 51 00 00 39 11 B9 80 D5 24 07 1F A3 AC | . .Q..9....$....
    00000020: 9F 0B EA 5D 10 92 00 0C 23 7F 61 61 61 0A 00 00 | ...]....#.aaa...
    00000030: 00 00 00 00 00 00 00 00 F5 8D CB 7C |  |  |  |  | ...........|

* case 4: when hw_strip_crc=1 and UDP data is
"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n"
(66 bytes, ie. > to 46):

    dump mbuf at 0x0x7effcc99ed80, phys=2db9ee00, buf_len=1920
      pkt_len=108, ol_flags=2, nb_segs=1, in_port=1
      segment at 0x0x7effcc99ed80, data=0x0x7effcc99ee80, data_len=108
      Dump data at [0x7effcc99ee80], len=108
    00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E.
    00000010: 00 5E 3B 4E 00 00 39 11 27 46 D5 24 07 1F A3 AC | .^;N..9.'F.$....
    00000020: 9F 0B EA 5D 10 92 00 4A 58 38 61 61 61 61 61 61 | ...]...JX8aaaaaa
    00000030: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000040: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000050: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa
    00000060: 61 61 61 61 61 61 61 61 61 61 61 0A |  |  |  |  | aaaaaaaaaaa.


There are a few things I don't understand:

* in case 1, why data_len = 60 and not 56, since hw_strip_crc=1? And
why the CRC is present at the end? (C6 63 C2 CF)
* case 2 seems totally normal: the CRC is not present
* for case 3, from what I understand, if hw_strip_crc is set to 0 the
ixgbe ddriver should remove the CRC
(http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n1399).
Why pkt len is 60 here? And why is the CRC present at the end (F5 8D
CB 7C)?
* case 4 seems ok, the CRC is absent, as it has been removed by the driver


The context : I am working on a NAT software using DPDK and I want to
offload as much as possible, hence the following code when emitting a
packet:

        udp_hdr->dgram_cksum = 0;
        pkt->ol_flags |= PKT_TX_UDP_CKSUM;
        udp_hdr->dgram_cksum = rte_ipv4_phdr_cksum(ipv4_header(pkt),
                                                   pkt->ol_flags);

When I'm sending a small UDP packet (small = below the minimal size of
the ethernet payload, ie. 46 bytes), the UDP checksum is always wrong.
The output of tcpdump:

        # small packet
        15:02:01.536009 IP (tos 0x0, ttl 54, id 59115, offset 0, flags
[none], proto UDP (17), length 31)
            213.36.7.31.59997 > 10.1.13.48.4242: [bad udp cksum 0x2cf1
-> 0xa612!] UDP, length 3
        # big packet
        15:11:27.240870 IP (tos 0x0, ttl 54, id 5823, offset 0, flags
[none], proto UDP (17), length 112)
            213.36.7.31.59997 > 10.1.13.48.4242: [udp sum ok] UDP, length 84


This is because I only rewrite some fields (source and/or destination
IP) of the rte_mbuf structure received before transmitting it, and the
UDP checksum calculation uses the CRC that hasn't been stripped
correctly as explained above.


My device is configured by calling rte_eth_dev_configure() with the
following eth_conf structure:

    static const struct rte_eth_conf eth_conf = {
        .link_speed=0,
        .link_duplex=0,
        .rxmode = {
            .mq_mode=ETH_MQ_RX_RSS,
            .jumbo_frame=0,
            .max_rx_pkt_len=0,
            .header_split=1,
            .split_hdr_size=64,
            .hw_ip_checksum=1,
            .hw_vlan_filter=1,
            .hw_vlan_strip=1,
            .hw_vlan_extend=0,
            .hw_strip_crc=1,
            .enable_scatter=1,
            .enable_lro=0,
        },
        .txmode = {
            .mq_mode=ETH_MQ_TX_NONE,
            .pvid=0,
            .hw_vlan_reject_tagged=0,
            .hw_vlan_reject_untagged=0,
            .hw_vlan_insert_pvid=0,
        },
        .lpbk_mode=0,
        .rx_adv_conf = {
            .rss_conf = {
                .rss_key=NULL,
                .rss_key_len=0,
                .rss_hf = ETH_RSS_PROTO_MASK,
            },
        },
    };


And some important details:

# lshw -class network
[...]
  *-network:0
       description: Ethernet controller
       product: 82599ES 10-Gigabit SFI/SFP+ Network Connection
[...]


# ./dpdk_nic_bind.py --status

Network devices using DPDK-compatible driver
============================================
0000:08:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection'
drv=igb_uio unused=
[...]


Thanks,
-- 
Julien Castets

Re: [dpdk-users] .hw_strip_crc weird behavior with small UDP packets

[Julien Castets]

It turns out the problem comes from a Cisco bug, not from DPDK.


First, I did some miscalculations, making me think the issue came from
DPDK — and it wasn't.

> * in case 1, why data_len = 60 and not 56, since hw_strip_crc=1? And
> why the CRC is present at the end? (C6 63 C2 CF)

data_len is 60 because the minimal Ethernet frame size is 64 bytes and
not 60, so after stripping the CRC by the hardware, the size is
reduced by 4. Which is normal. The last four bytes don't correspond to
the CRC though and should be 0 (explained below).

> * case 2 seems totally normal: the CRC is not present

Yes, that's normal

> * for case 3, from what I understand, if hw_strip_crc is set to 0 the
> ixgbe ddriver should remove the CRC
> (http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n1399).
> Why pkt len is 60 here? And why is the CRC present at the end (F5 8D
> CB 7C)?

Exactly the same case than case 1. data_len is normal (but here the
CRC is stripped by the driver and not by the hardware), and as in case
1, the last four bytes should be 0 but aren't.

> * case 4 seems ok, the CRC is absent, as it has been removed by the driver

Yes, that's normal.


The problem is my network topology: my NAT software receives traffic
from a Cisco Nexus 9000 which is buggy. When the Nexus untags a packet
of 64 bytes (the minimal Ethernet size), it adds an extra four bytes
padding before the CRC — which is expected. However, those bytes
should always be zeros, but here they are random, hence my checksum
issue.


Sorry for the noise,
-- 
Julien Castets