From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from rcdn-iport-3.cisco.com (rcdn-iport-3.cisco.com [173.37.86.74]) by dpdk.org (Postfix) with ESMTP id 162D11B1A3 for ; Wed, 6 Dec 2017 12:21:52 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cisco.com; i=@cisco.com; l=12730; q=dns/txt; s=iport; t=1512559313; x=1513768913; h=from:to:cc:subject:date:message-id:references: in-reply-to:content-transfer-encoding:mime-version; bh=/iUtxvW58JVNmgQKU48nIfjvdC7RadH3U3d+5mUevCE=; b=JrOlXK5d7qR7k9iGqQyT86rmJc2oLni3jYc1VUx6xDdPR9WlEqm4SOhP IwZti9hq0xC+bJIOP0pNvxaDSqbDMHtz08FBAPDyCm3gj/3tMnmn+m5po nzz+Uanp3PfZSh/+Hbt72JOt5DlWrUZgTQIilLzIVpu/0xlZnX8kDUlX7 0=; X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0DtAABX0ida/4ENJK1dGQEBAQEBAQEBA?= =?us-ascii?q?QEBAQcBAQEBAYM9Zm4nB44bjnuBfZcFghUKG4UgAoVSPxgBAQEBAQEBAQFrKIU?= =?us-ascii?q?iAQEBAQMnEz8MBAIBCBEDAQEBHwkHMhQJCAIEDgUIihurXjqKUgEBAQEBAQEBA?= =?us-ascii?q?QEBAQEBAQEBAQEBAR2FVoFWgWmCHYEOixkFikWYOAKHdI0ak2SWJgIRGQGBOQE?= =?us-ascii?q?fOSaBKG8VgmMJgkkcGYFOeIkNAYEUAQEB?= X-IronPort-AV: E=Sophos;i="5.45,367,1508803200"; d="scan'208";a="318685914" Received: from alln-core-9.cisco.com ([173.36.13.129]) by rcdn-iport-3.cisco.com with ESMTP/TLS/DHE-RSA-AES256-SHA; 06 Dec 2017 11:21:51 +0000 Received: from XCH-RTP-016.cisco.com (xch-rtp-016.cisco.com [64.101.220.156]) by alln-core-9.cisco.com (8.14.5/8.14.5) with ESMTP id vB6BLpr5010646 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=FAIL); Wed, 6 Dec 2017 11:21:51 GMT Received: from xch-rtp-017.cisco.com (64.101.220.157) by XCH-RTP-016.cisco.com (64.101.220.156) with Microsoft SMTP Server (TLS) id 15.0.1320.4; Wed, 6 Dec 2017 06:21:50 -0500 Received: from xch-rtp-017.cisco.com ([64.101.220.157]) by XCH-RTP-017.cisco.com ([64.101.220.157]) with mapi id 15.00.1320.000; Wed, 6 Dec 2017 06:21:50 -0500 From: "Hanoch Haim (hhaim)" To: "Xing, Beilei" , "Wu, Jingjing" CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF Thread-Index: AdNiEt0UIw83Y/69THKQx+TvljuhYgDA+ARgAMu8aBAAANIg0AAD3UOgAGiGC4AAAeXjwAEfPR0QAADcHyA= Date: Wed, 6 Dec 2017 11:21:50 +0000 Message-ID: <04e84e3f795949e0b3fb579c6ecb3bcd@XCH-RTP-017.cisco.com> References: <3f57eb6982af4bb9aae69bce67233d89@XCH-RTP-017.cisco.com> <9BB6961774997848B5B42BEC655768F810EC8AC3@SHSMSX103.ccr.corp.intel.com> <9BB6961774997848B5B42BEC655768F810EC8D87@SHSMSX103.ccr.corp.intel.com> <9BB6961774997848B5B42BEC655768F810ECB645@SHSMSX103.ccr.corp.intel.com> <94479800C636CB44BD422CB454846E013207A2F6@SHSMSX101.ccr.corp.intel.com> In-Reply-To: <94479800C636CB44BD422CB454846E013207A2F6@SHSMSX101.ccr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-transport-fromentityheader: Hosted x-originating-ip: [64.103.125.71] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 11:21:53 -0000 Hi Beilei, Thanks for looking into this.=20 I'm testing it with TRex (tx->rx) and not (rx->tx) in a bit different confi= guration and there are packets with ~40usec latency (Tx/Rx) without my patc= h So I assume the latency comes from a different place due to this change (IN= TR) TRex command: $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -= c 1 I'm testing it in loopback and in this configuration there are two Tx queue= s and 2 Rx queues RSS is not configured all Rx traffic goes to Rx0 Core0 -------- Tx0 - TCP flows Rx0 - TCP flows=20 Core1 -------- Tx1 - latency flows (ICMP) - low rate 1KPPS=20 Rx1 - latency flows (ICMP) - match a rule of TOS/TTL to redirect to this Rx= 1 queue. Qos configuration=20 ------------------------ ETS, Tx1 is configured as low latency/high priority =20 Throughput of Tx0/Rx0 is ~10Gb/average packet size =3D~700B The latency is measured betwean Tx1 to Rx1=20 Thanks, Hanoh -----Original Message----- From: Xing, Beilei [mailto:beilei.xing@intel.com]=20 Sent: Wednesday, December 06, 2017 1:00 PM To: Wu, Jingjing; Hanoch Haim (hhaim) Cc: dev@dpdk.org Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttlin= g setting in PF Hi, I tested Rx latency with testpmd and ixia, found that the interval configur= ation works on X710. With the default configuration, the latency is about 32us.=20 When RTE_LIBRTE_I40E_ITR_INTERVAL=3D0, the max latency is < 8us. When RTE_LIBRTE_I40E_ITR_INTERVAL=3D8160, the max latency is about 8ms. My test steps: 1. connect a X710 port with ixia, and bind the port to igb_uio 2. run testp= md and start io forwding 3. send a burst (5 packets) with ixia to X710 port= 4. check the latency on ixia. Best Regards, Beilei > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wu, Jingjing > Sent: Friday, December 1, 2017 1:47 AM > To: Hanoch Haim (hhaim) > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt=20 > throttling setting in PF >=20 > Hi, Hanoch >=20 > Thanks a lot for the trying. Are you using igb_uio to bind the device? >=20 > I guess it would be because that fix is not complete, the overlap with =20 > Rx interrupt mode is not considered. We will look into it. > And it would be great if you can have a try on vfio_pci cases. >=20 > Thanks > Jingjing >=20 > From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] > Sent: Friday, December 1, 2017 12:48 AM > To: Wu, Jingjing > Cc: dev@dpdk.org; Hanoch Haim (hhaim) > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt=20 > throttling setting in PF >=20 > Hi Jingjing, > I did that and see the results, It does not work as expected >=20 > TRex command: > $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000=20 > --astf -c 1 >=20 >=20 >=20 > 1) with the issue (without the patch) > *itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | *#define=20 > RTE_LIBRTE_I40E_ITR_INTERVAL -1 >=20 >=20 > latency : 40usec >=20 > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter= max > window > | , , , , average , max , (usec) > =20 > ---------------------------------------------------------------------- > ----------------------- > ------------------- > 0 | 5603, 5603, 0, 0, 6 , 43, 1 = | 0 0 0 0 41 41 34 40 41 > 33 43 8 8 > 1 | 5603, 5603, 0, 0, 12 , 44, 24 = | 0 0 0 0 39 41 41 34 40 > 33 33 8 8 > 2 | 5603, 5603, 0, 0, 8 , 43, 5 = | 0 0 0 0 38 41 42 40 40 > 40 42 9 8 > 3 | 5603, 5603, 0, 0, 6 , 43, 1 = | 0 0 0 0 36 41 34 42 43 > 8 35 40 41 > *** TRex is shutting down - cause: 'CTRL + C detected' >=20 > 2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4 >=20 > itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define=20 > RTE_LIBRTE_I40E_ITR_INTERVAL 4 >=20 > latency : 40usec >=20 > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter= max > window > | , , , , average , max , (usec) > =20 > ---------------------------------------------------------------------- > ----------------------- > ------------------- > 0 | 5034, 5034, 0, 0, 10 , 42, 0 = | 0 0 0 0 0 42 40 38 24 > 32 23 24 23 > 1 | 5034, 5034, 0, 0, 8 , 43, 0 = | 0 0 0 0 0 43 37 38 19 > 20 30 18 21 > 2 | 5034, 5034, 0, 0, 8 , 45, 0 = | 0 0 0 0 0 37 40 41 40 > 41 41 40 45 > 3 | 5034, 5034, 0, 0, 8 , 48, 0 = | 0 0 0 0 0 42 36 43 44 > 43 43 44 48 > * >=20 >=20 > 3) RTE_LIBRTE_I40E_ITR_INTERVAL 0 >=20 > itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define=20 > RTE_LIBRTE_I40E_ITR_INTERVAL 0 >=20 > latency : 40usec >=20 > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter= max > window > | , , , , average , max , (usec) > =20 > ---------------------------------------------------------------------- > ----------------------- > ------------------- > 0 | 5034, 5034, 0, 0, 10 , 42, 0 = | 0 0 0 0 0 42 40 38 24 > 32 23 24 23 > 1 | 5034, 5034, 0, 0, 8 , 43, 0 = | 0 0 0 0 0 43 37 38 19 > 20 30 18 21 > 2 | 5034, 5034, 0, 0, 8 , 45, 0 = | 0 0 0 0 0 37 40 41 40 > 41 41 40 45 > 3 | 5034, 5034, 0, 0, 8 , 48, 0 = | 0 0 0 0 0 42 36 43 44 > 43 43 44 48 > * >=20 > 4) TRex patch issue solved >=20 > I40E_QINT_RQCTL_ITR_INDX_MASK > #define RTE_LIBRTE_I40E_ITR_INTERVAL -1 >=20 > latency : 8usec >=20 > -Latency stats enabled > Cpu Utilization : 0.1 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter= max > window > | , , , , average , max , (usec) > =20 > ---------------------------------------------------------------------- > ----------------------- > ------------------- > 0 | 9501, 9501, 0, 0, 7 , 21, 0 = | 21 9 9 12 9 9 14 12 8 9 > 9 9 8 > 1 | 9501, 9501, 0, 0, 7 , 25, 0 = | 22 8 9 12 9 9 15 12 8 8 > 9 9 8 > 2 | 9501, 9501, 0, 0, 6 , 26, 0 = | 22 9 10 9 10 10 15 13 8 > 9 9 9 8 > 3 | 9501, 9501, 0, 0, 6 , 32, 0 = | 22 8 9 9 9 9 15 12 8 7 > 9 9 9 >=20 > Thanks, > Hanoh >=20 > From: Wu, Jingjing [mailto:jingjing.wu@intel.com] > Sent: Tuesday, November 28, 2017 5:01 PM > To: Hanoch Haim (hhaim) > Cc: dev@dpdk.org > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt=20 > throttling setting in PF >=20 > Hi, Hanoch >=20 > If DPDK PF, the commit affects that because it introduces an argument > (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx=20 > with default value 32us. >=20 > If you'd like to get the descriptor write back immediately, you can=20 > set "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=3D0" in config/common_base file. >=20 > Or you can just change the definition of=20 > I40E_QUEUE_ITR_INTERVAL_DEFAULT like: > #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */ >=20 > Thanks > Jingjing >=20 > From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] > Sent: Tuesday, November 28, 2017 9:14 PM > To: Wu, Jingjing > > Cc: dev@dpdk.org; Hanoch Haim (hhaim)=20 > > > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt=20 > throttling setting in PF >=20 > Hi Jingjing, >=20 > 1. The issue is with DPDK PF. >=20 > 2. The rate is high ~10gb, one DP core, one latency core. >=20 > 3. The fix is here >=20 > /* Bind all RX queues to allocated MSIX interrupt */ > for (i =3D 0; i < nb_queue; i++) { > val =3D (msix_vect << I40E_QINT_RQCTL_MSI= X_INDX_SHIFT) | > #ifdef TREX_PATCH > =20 > I40E_QINT_RQCTL_ITR_INDX_MASK | << low latency 11b =3D NoITR > #else > itr_idx <<=20 > I40E_QINT_RQCTL_ITR_INDX_SHIFT | << high spkies > #endif >=20 > I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val); >=20 > The Interrupt Throttling ITR is configure using a different setting=20 > using a different register here : >=20 >=20 > 4. The ITR_INTERVAL is 32 usec and it affect a different PF register >=20 > #define I40E_ITR_INDEX_DEFAULT 0 > #define I40E_ITR_INDEX_NONE 3 > #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */ >=20 >=20 > 5. My question is why the VF configuration affects PF INT_INTERVAL ?= Can > I remove my patch and fix this latency issue in the different way? >=20 > Thanks, > Hanoh >=20 > From: Wu, Jingjing [mailto:jingjing.wu@intel.com] > Sent: Tuesday, November 28, 2017 2:50 PM > To: Hanoch Haim (hhaim); dev@dpdk.org > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt=20 > throttling setting in PF >=20 > Hi, Hanoch >=20 > Are you talking about i40 VF's latency? And you are using DPDK PF as=20 > host driver? >=20 > In this case, we are setting the Interrupt Throttling (ITR) to be maximum= . > That is to say, if the packet rate is very slow , the receive=20 > descriptor is written back when ITR timeout, otherwise it is written=20 > back when cache line is full (4 descriptors/packets). I think that's why = you saw the latency is varying. >=20 > If we change the ITR to minor, then huge number of interrupts will=20 > coming to core which impact performance. >=20 >=20 > Thanks > Jingjing >=20 > From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] > Sent: Friday, November 24, 2017 7:25 PM > To: dev@dpdk.org > Cc: Wu, Jingjing > > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt=20 > throttling setting in PF >=20 > Re-sending >=20 > Hanoh >=20 > From: Hanoch Haim (hhaim) > Sent: Monday, November 20, 2017 5:19 PM > To: dev@dpdk.org > Cc: Wu, Jingjing=20 > (jingjing.wu@intel.com); > Hanoch Haim (hhaim) > Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt=20 > throttling setting in PF >=20 > Hi All, > While integrating dpdk17.11 into TRex latest code a new latency issue=20 > is observed (i40e is very sensitive because it has very good=20 > resolution due to Qos configuration). > git bitsec found the following commit. > With this commit we observe high spikes of Rx latency (~40usec) vs (~8use= c). > Any idea why? > I can send how to reproduce this, it is very simple. >=20 > cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit=20 > commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e > Author: Jingjing Wu > > Date: Thu Aug 24 09:57:51 2017 +0800 >=20 > net/i40e: fix interrupt throttling setting in PF >=20 > As no matter the PF host driver is DPDK or other kernel drivers, > they are sharing the same virtchnnl interfaces to communicate to VFs. > To follow the generic interface, DPDK PF need to set Interrupt > Throttling (ITR) index according to the rxitr_idx from virtchnnl > instead of ITR_NONE. >=20 > Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support=20 > Linux > VF") > Cc: stable@dpdk.org >=20 > Signed-off-by: Jingjing Wu > > >=20 >=20 >=20 > Thanks, > Hanoh