From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f45.google.com (mail-wm0-f45.google.com [74.125.82.45]) by dpdk.org (Postfix) with ESMTP id EBFA395CC for ; Mon, 1 Feb 2016 17:51:49 +0100 (CET) Received: by mail-wm0-f45.google.com with SMTP id p63so80472521wmp.1 for ; Mon, 01 Feb 2016 08:51:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=6DxlfNVTWPpDgqCecdpDWSan5ynXz9F72rqm7Ig8fig=; b=CCxWKKKArUwAoqYk/XOzpRaOQ/iIDC54kE48PSvUVGrTFWpS/vdbl5ztY6FNfd3ZOC Vy2Ph4l1YUv1XnKYxh78edToLGK1LuWZUnJCHAVXmuuWgImVFsxY5OU+PvoM8b5NaDlm pmw4RIF3C58hYPG+/LwfYW1WPAiQlHTa7k8lSKw1K5DgKFBiWipTfu4gG9kG/gixubO2 8lDZPHago/TKPENBgbY3ksPdfV3FosPTpETI8PjfPlqsxNOpttvUH6gbw7owGsmr7/mk iThJ50N1o017/w/sb6FW6IvS98ed8TiKIAl4212EKv8srVFdC/7I/jeQCxbOWhq+aUsp //pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:content-type; bh=6DxlfNVTWPpDgqCecdpDWSan5ynXz9F72rqm7Ig8fig=; b=BZG0+50Zjh0zEh4N9mLkwQgP4NqGRC0Ic32oVL50gRbrFE1DIsU0qn+1WPBrRQ67MF 6tuELVcOtxcdaBJ04TKgUNhYsBp5Y/z2nf6laVjnOwD2qNfIS5AyJxlFnok+7GJrCaqn BncWxOPtvXMfSZcXZfsfeX51RdzfwHfoAXVGaFmpbpr7Y+cWS/GW24ZmuwOPpdWraI2e z/NjAyR8YmpiZW+NHGjSOZlMT8vVDfmmbi5Y4hM5hsDz7zTuV8GlfeHDJ03UGQvTCHsN JyIqRhpzEFCdUGKMYrf8QEVsWfOcZKyM4NIIZKQSUr/h1etDxKtkLNChbXQRXXgiwZyb zTKg== X-Gm-Message-State: AG10YOTT0bSEdcJwPlyHVwXxX2D0tTplFqBs5VPk+Juc/HP3XnrLPs2/aoZo0ELlJ8QFJLD/fzdy/Y8AB1uJzw== MIME-Version: 1.0 X-Received: by 10.28.30.198 with SMTP id e189mr9748910wme.60.1454345509650; Mon, 01 Feb 2016 08:51:49 -0800 (PST) Sender: castet.j@gmail.com Received: by 10.194.2.106 with HTTP; Mon, 1 Feb 2016 08:51:49 -0800 (PST) Date: Mon, 1 Feb 2016 17:51:49 +0100 X-Google-Sender-Auth: x7pShrL5_3-wqMwJb4lH6i8DE_A Message-ID: From: Julien Castets To: users@dpdk.org Content-Type: text/plain; charset=UTF-8 Subject: [dpdk-users] .hw_strip_crc weird behavior with small UDP packets X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Feb 2016 16:51:50 -0000 Hi, I dump the packets received with rte_pktmbuf_dump(stderr, pkt, pkt->data_len) and I have a strange behavior related to CRC stripping with small UDP packets. By "small", I mean below the minimal ethernet payload size, ie. 46 bytes. * case 1: when my device is configured with hw_strip_crc=1 and the UDP data is "aaa\n" (4 bytes), my function outputs: dump mbuf at 0x0x7fe93bdc47c0, phys=2dbc4840, buf_len=1920 pkt_len=60, ol_flags=2, nb_segs=1, in_port=1 segment at 0x0x7fe93bdc47c0, data=0x0x7fe93bdc48c0, data_len=60 Dump data at [0x7fe93bdc48c0], len=60 00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E. 00000010: 00 20 26 D5 00 00 39 11 3B FD D5 24 07 1F A3 AC | . &...9.;..$.... 00000020: 9F 0B EA 5D 10 92 00 0C 23 7F 61 61 61 0A 00 00 | ...]....#.aaa... 00000030: 00 00 00 00 00 00 00 00 C6 63 C2 CF | | | | | .........c.. * case 2: when hw_strip_crc=1 and UDP data is "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n" (66 bytes, ie. > to 46): dump mbuf at 0x0x7fe93bdd4fc0, phys=2dbd5040, buf_len=1920 pkt_len=108, ol_flags=2, nb_segs=1, in_port=1 segment at 0x0x7fe93bdd4fc0, data=0x0x7fe93bdd50c0, data_len=108 Dump data at [0x7fe93bdd50c0], len=108 00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E. 00000010: 00 5E F3 BA 00 00 39 11 6E D9 D5 24 07 1F A3 AC | .^....9.n..$.... 00000020: 9F 0B EA 5D 10 92 00 4A 58 38 61 61 61 61 61 61 | ...]...JX8aaaaaa 00000030: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa 00000040: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa 00000050: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa 00000060: 61 61 61 61 61 61 61 61 61 61 61 0A | | | | | aaaaaaaaaaa. * case 3: when hw_strip_crc=0 and data is "aaa\n" (4 bytes): dump mbuf at 0x0x7effcc996140, phys=2db961c0, buf_len=1920 pkt_len=60, ol_flags=2, nb_segs=1, in_port=1 segment at 0x0x7effcc996140, data=0x0x7effcc996240, data_len=60 Dump data at [0x7effcc996240], len=60 00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E. 00000010: 00 20 A9 51 00 00 39 11 B9 80 D5 24 07 1F A3 AC | . .Q..9....$.... 00000020: 9F 0B EA 5D 10 92 00 0C 23 7F 61 61 61 0A 00 00 | ...]....#.aaa... 00000030: 00 00 00 00 00 00 00 00 F5 8D CB 7C | | | | | ...........| * case 4: when hw_strip_crc=1 and UDP data is "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\n" (66 bytes, ie. > to 46): dump mbuf at 0x0x7effcc99ed80, phys=2db9ee00, buf_len=1920 pkt_len=108, ol_flags=2, nb_segs=1, in_port=1 segment at 0x0x7effcc99ed80, data=0x0x7effcc99ee80, data_len=108 Dump data at [0x7effcc99ee80], len=108 00000000: 38 EA A7 92 70 28 58 97 BD 4F 6F C7 08 00 45 00 | 8...p(X..Oo...E. 00000010: 00 5E 3B 4E 00 00 39 11 27 46 D5 24 07 1F A3 AC | .^;N..9.'F.$.... 00000020: 9F 0B EA 5D 10 92 00 4A 58 38 61 61 61 61 61 61 | ...]...JX8aaaaaa 00000030: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa 00000040: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa 00000050: 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 | aaaaaaaaaaaaaaaa 00000060: 61 61 61 61 61 61 61 61 61 61 61 0A | | | | | aaaaaaaaaaa. There are a few things I don't understand: * in case 1, why data_len = 60 and not 56, since hw_strip_crc=1? And why the CRC is present at the end? (C6 63 C2 CF) * case 2 seems totally normal: the CRC is not present * for case 3, from what I understand, if hw_strip_crc is set to 0 the ixgbe ddriver should remove the CRC (http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx.c#n1399). Why pkt len is 60 here? And why is the CRC present at the end (F5 8D CB 7C)? * case 4 seems ok, the CRC is absent, as it has been removed by the driver The context : I am working on a NAT software using DPDK and I want to offload as much as possible, hence the following code when emitting a packet: udp_hdr->dgram_cksum = 0; pkt->ol_flags |= PKT_TX_UDP_CKSUM; udp_hdr->dgram_cksum = rte_ipv4_phdr_cksum(ipv4_header(pkt), pkt->ol_flags); When I'm sending a small UDP packet (small = below the minimal size of the ethernet payload, ie. 46 bytes), the UDP checksum is always wrong. The output of tcpdump: # small packet 15:02:01.536009 IP (tos 0x0, ttl 54, id 59115, offset 0, flags [none], proto UDP (17), length 31) 213.36.7.31.59997 > 10.1.13.48.4242: [bad udp cksum 0x2cf1 -> 0xa612!] UDP, length 3 # big packet 15:11:27.240870 IP (tos 0x0, ttl 54, id 5823, offset 0, flags [none], proto UDP (17), length 112) 213.36.7.31.59997 > 10.1.13.48.4242: [udp sum ok] UDP, length 84 This is because I only rewrite some fields (source and/or destination IP) of the rte_mbuf structure received before transmitting it, and the UDP checksum calculation uses the CRC that hasn't been stripped correctly as explained above. My device is configured by calling rte_eth_dev_configure() with the following eth_conf structure: static const struct rte_eth_conf eth_conf = { .link_speed=0, .link_duplex=0, .rxmode = { .mq_mode=ETH_MQ_RX_RSS, .jumbo_frame=0, .max_rx_pkt_len=0, .header_split=1, .split_hdr_size=64, .hw_ip_checksum=1, .hw_vlan_filter=1, .hw_vlan_strip=1, .hw_vlan_extend=0, .hw_strip_crc=1, .enable_scatter=1, .enable_lro=0, }, .txmode = { .mq_mode=ETH_MQ_TX_NONE, .pvid=0, .hw_vlan_reject_tagged=0, .hw_vlan_reject_untagged=0, .hw_vlan_insert_pvid=0, }, .lpbk_mode=0, .rx_adv_conf = { .rss_conf = { .rss_key=NULL, .rss_key_len=0, .rss_hf = ETH_RSS_PROTO_MASK, }, }, }; And some important details: # lshw -class network [...] *-network:0 description: Ethernet controller product: 82599ES 10-Gigabit SFI/SFP+ Network Connection [...] # ./dpdk_nic_bind.py --status Network devices using DPDK-compatible driver ============================================ 0000:08:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection' drv=igb_uio unused= [...] Thanks, -- Julien Castets