From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-fw-33001.amazon.com (smtp-fw-33001.amazon.com [207.171.189.228]) by dpdk.org (Postfix) with ESMTP id D5AB7DE4 for ; Fri, 1 Nov 2013 15:53:37 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=xingbow@amazon.com; q=dns/txt; s=amazon201209; t=1383317670; x=1414853670; h=from:to:cc:subject:date:message-id:in-reply-to: content-id:content-transfer-encoding:mime-version; bh=JeLNwKOBeOX/RBIQ9MliH7jVw7TyXA57w/gJuS7NcBw=; b=ZDZyqAZZMTuS0ehvHVWuteCT7MxEblIarujR9/Fy0tvwKcYb5F9Peuky vqAbdmgUi7ZzAmBLqFYVXNlZHFU6Y64NFRgP627fiDNR0o1vzCsa/iShA H4Vdj4qw/ee69v3P8cKIIABvajQ4kB5aa38lyeEyU8pexSlHIsZuFvb4N E=; X-IronPort-AV: E=Sophos;i="4.93,617,1378857600"; d="scan'208";a="646629149" Received: from smtp-in-1104.vdc.amazon.com ([10.140.10.25]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 01 Nov 2013 14:54:23 +0000 Received: from ex10-hub-31005.ant.amazon.com (ex10-hub-31005.sea31.amazon.com [10.185.176.12]) by smtp-in-1104.vdc.amazon.com (8.14.7/8.14.7) with ESMTP id rA1EsFKI000579 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=OK); Fri, 1 Nov 2013 14:54:22 GMT Received: from EX10-MBX-31007.ant.amazon.com ([fe80::dc2d:ebf:d4a1:fac]) by ex10-hub-31005.ant.amazon.com ([::1]) with mapi id 14.02.0342.003; Fri, 1 Nov 2013 07:54:21 -0700 From: "Wang, Shawn" To: Alexander Belyakov Thread-Topic: [dpdk-dev] Surprisingly high TCP ACK packets drop counter Thread-Index: AQHO1whjRxNd259ED0ycA7IZkDyueZoQdm4A Date: Fri, 1 Nov 2013 14:54:20 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.3.8.130913 x-originating-ip: [10.184.49.66] Content-Type: text/plain; charset="iso-8859-1" Content-ID: <9BAD35530CB6B74E86AAE1843E8DA3D7@ant.amazon.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Precedence: Bulk Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Surprisingly high TCP ACK packets drop counter X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Nov 2013 14:53:38 -0000 Hi: We had the same problem before. It turned out that RSC (receive side coalescing) is enabled by default in DPDK. So we write this na=EFve patch t= o disable it. This patch is based on DPDK 1.3. Not sure 1.5 has changed it or not. After this patch, ACK rate should go back to 14.5Mpps. For details, you can refer to Intel=AE 82599 10 GbE Controller Datasheet. (7.11 Receive Side Coalescing). From: xingbow Date: Wed, 21 Aug 2013 11:35:23 -0700 Subject: [PATCH] Disable RSC in ixgbe_dev_rx_init function in file ixgbe_rxtx.c =20 --- DPDK/lib/librte_pmd_ixgbe/ixgbe/ixgbe_type.h | 2 +- DPDK/lib/librte_pmd_ixgbe/ixgbe_rxtx.c | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletion(-) =20 diff --git a/DPDK/lib/librte_pmd_ixgbe/ixgbe/ixgbe_type.h b/DPDK/lib/librte_pmd_ixgbe/ixgbe/ixgbe_type.h index 7fffd60..f03046f 100644 --- a/DPDK/lib/librte_pmd_ixgbe/ixgbe/ixgbe_type.h +++ b/DPDK/lib/librte_pmd_ixgbe/ixgbe/ixgbe_type.h @@ -1930,7 +1930,7 @@ enum { #define IXGBE_RFCTL_ISCSI_DIS 0x00000001 #define IXGBE_RFCTL_ISCSI_DWC_MASK 0x0000003E #define IXGBE_RFCTL_ISCSI_DWC_SHIFT 1 -#define IXGBE_RFCTL_RSC_DIS 0x00000010 +#define IXGBE_RFCTL_RSC_DIS 0x00000020 #define IXGBE_RFCTL_NFSW_DIS 0x00000040 #define IXGBE_RFCTL_NFSR_DIS 0x00000080 #define IXGBE_RFCTL_NFS_VER_MASK 0x00000300 diff --git a/DPDK/lib/librte_pmd_ixgbe/ixgbe_rxtx.c b/DPDK/lib/librte_pmd_ixgbe/ixgbe_rxtx.c index 07830b7..ba6e05d 100755 --- a/DPDK/lib/librte_pmd_ixgbe/ixgbe_rxtx.c +++ b/DPDK/lib/librte_pmd_ixgbe/ixgbe_rxtx.c @@ -3007,6 +3007,7 @@ ixgbe_dev_rx_init(struct rte_eth_dev *dev) uint64_t bus_addr; uint32_t rxctrl; uint32_t fctrl; + uint32_t rfctl; uint32_t hlreg0; uint32_t maxfrs; uint32_t srrctl; @@ -3033,6 +3034,12 @@ ixgbe_dev_rx_init(struct rte_eth_dev *dev) fctrl |=3D IXGBE_FCTRL_PMCF; IXGBE_WRITE_REG(hw, IXGBE_FCTRL, fctrl); =20 + /* Disable RSC */ + RTE_LOG(INFO, PMD, "Disable RSC\n"); + rfctl =3D IXGBE_READ_REG(hw, IXGBE_RFCTL); + rfctl |=3D IXGBE_RFCTL_RSC_DIS; + IXGBE_WRITE_REG(hw, IXGBE_RFCTL, rfctl); + /* * Configure CRC stripping, if any. */ --=20 Thanks. Wang, Xingbo On 11/1/13 6:43 AM, "Alexander Belyakov" wrote: >Hello, > >we have simple test application on top of DPDK which sole purpose is to >forward as much packets as possible. Generally we easily achieve 14.5Mpps >with two 82599EB (one as input and one as output). The only suprising >exception is forwarding pure TCP ACK flood when performace always drops to >approximately 7Mpps. > >For simplicity consider two different types of traffic: >1) TCP SYN flood is forwarded at 14.5Mpps rate, >2) pure TCP ACK flood is forwarded only at 7Mpps rate. > >Both SYN and ACK packets have exactly the same length. > >It is worth to mention, this forwarding application looks at Ethernet and >IP headers, but never deals with L4 headers. > >We tracked down issue to RX circuit. To be specific, there are 4 RX queues >initialized on input port and rte_eth_stats_get() shows uniform packet >distribution (q_ipackets) among them, while q_errors remain zero for all >queues. The only drop counter quickly increasing in the case of pure ACK >flood is ierrors, while rx_nombuf remains zero. > >We tried different kinds of traffic generators, but always got the same >result: 7Mpps (instead of expected 14Mpps) for TCP packets with ACK flag >bit set while all other flag bits dropped. Source IPs and ports are >selected randomly. > >Please let us know if anyone is aware of such strange behavior and where >should we look at to narrow down the problem. > >Thanks in advance, >Alexander Belyakov