From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 6D61B1B1A4 for ; Wed, 6 Dec 2017 12:00:33 +0100 (CET) Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Dec 2017 03:00:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.45,367,1508828400"; d="scan'208";a="10097003" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by fmsmga004.fm.intel.com with ESMTP; 06 Dec 2017 03:00:32 -0800 Received: from fmsmsx153.amr.corp.intel.com (10.18.125.6) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 6 Dec 2017 03:00:32 -0800 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by FMSMSX153.amr.corp.intel.com (10.18.125.6) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 6 Dec 2017 03:00:31 -0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.159]) by shsmsx102.ccr.corp.intel.com ([169.254.2.175]) with mapi id 14.03.0319.002; Wed, 6 Dec 2017 19:00:30 +0800 From: "Xing, Beilei" To: "Wu, Jingjing" , "Hanoch Haim (hhaim)" CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF Thread-Index: AdNiEt0UIw83Y/69THKQx+TvljuhYgDA+ARgAMu8aBAAANIg0AAD3UOgAGiGC4AAAeXjwAEfPR0Q Date: Wed, 6 Dec 2017 11:00:29 +0000 Message-ID: <94479800C636CB44BD422CB454846E013207A2F6@SHSMSX101.ccr.corp.intel.com> References: <3f57eb6982af4bb9aae69bce67233d89@XCH-RTP-017.cisco.com> <9BB6961774997848B5B42BEC655768F810EC8AC3@SHSMSX103.ccr.corp.intel.com> <9BB6961774997848B5B42BEC655768F810EC8D87@SHSMSX103.ccr.corp.intel.com> <9BB6961774997848B5B42BEC655768F810ECB645@SHSMSX103.ccr.corp.intel.com> In-Reply-To: <9BB6961774997848B5B42BEC655768F810ECB645@SHSMSX103.ccr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Dec 2017 11:00:34 -0000 Hi, I tested Rx latency with testpmd and ixia, found that the interval configur= ation works on X710. With the default configuration, the latency is about 32us.=20 When RTE_LIBRTE_I40E_ITR_INTERVAL=3D0, the max latency is < 8us. When RTE_LIBRTE_I40E_ITR_INTERVAL=3D8160, the max latency is about 8ms. My test steps: 1. connect a X710 port with ixia, and bind the port to igb_uio 2. run testpmd and start io forwding 3. send a burst (5 packets) with ixia to X710 port 4. check the latency on ixia. Best Regards, Beilei > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wu, Jingjing > Sent: Friday, December 1, 2017 1:47 AM > To: Hanoch Haim (hhaim) > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt throttl= ing > setting in PF >=20 > Hi, Hanoch >=20 > Thanks a lot for the trying. Are you using igb_uio to bind the device? >=20 > I guess it would be because that fix is not complete, the overlap with R= x > interrupt mode is not considered. We will look into it. > And it would be great if you can have a try on vfio_pci cases. >=20 > Thanks > Jingjing >=20 > From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] > Sent: Friday, December 1, 2017 12:48 AM > To: Wu, Jingjing > Cc: dev@dpdk.org; Hanoch Haim (hhaim) > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttl= ing > setting in PF >=20 > Hi Jingjing, > I did that and see the results, It does not work as expected >=20 > TRex command: > $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -= c 1 >=20 >=20 >=20 > 1) with the issue (without the patch) > *itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | *#define > RTE_LIBRTE_I40E_ITR_INTERVAL -1 >=20 >=20 > latency : 40usec >=20 > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter= max > window > | , , , , average , max , (usec) > ------------------------------------------------------------------------= --------------------- > ------------------- > 0 | 5603, 5603, 0, 0, 6 , 43, 1 = | 0 0 0 0 41 41 34 40 41 > 33 43 8 8 > 1 | 5603, 5603, 0, 0, 12 , 44, 24 = | 0 0 0 0 39 41 41 34 40 > 33 33 8 8 > 2 | 5603, 5603, 0, 0, 8 , 43, 5 = | 0 0 0 0 38 41 42 40 40 > 40 42 9 8 > 3 | 5603, 5603, 0, 0, 6 , 43, 1 = | 0 0 0 0 36 41 34 42 43 > 8 35 40 41 > *** TRex is shutting down - cause: 'CTRL + C detected' >=20 > 2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4 >=20 > itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define > RTE_LIBRTE_I40E_ITR_INTERVAL 4 >=20 > latency : 40usec >=20 > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter= max > window > | , , , , average , max , (usec) > ------------------------------------------------------------------------= --------------------- > ------------------- > 0 | 5034, 5034, 0, 0, 10 , 42, 0 = | 0 0 0 0 0 42 40 38 24 > 32 23 24 23 > 1 | 5034, 5034, 0, 0, 8 , 43, 0 = | 0 0 0 0 0 43 37 38 19 > 20 30 18 21 > 2 | 5034, 5034, 0, 0, 8 , 45, 0 = | 0 0 0 0 0 37 40 41 40 > 41 41 40 45 > 3 | 5034, 5034, 0, 0, 8 , 48, 0 = | 0 0 0 0 0 42 36 43 44 > 43 43 44 48 > * >=20 >=20 > 3) RTE_LIBRTE_I40E_ITR_INTERVAL 0 >=20 > itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define > RTE_LIBRTE_I40E_ITR_INTERVAL 0 >=20 > latency : 40usec >=20 > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter= max > window > | , , , , average , max , (usec) > ------------------------------------------------------------------------= --------------------- > ------------------- > 0 | 5034, 5034, 0, 0, 10 , 42, 0 = | 0 0 0 0 0 42 40 38 24 > 32 23 24 23 > 1 | 5034, 5034, 0, 0, 8 , 43, 0 = | 0 0 0 0 0 43 37 38 19 > 20 30 18 21 > 2 | 5034, 5034, 0, 0, 8 , 45, 0 = | 0 0 0 0 0 37 40 41 40 > 41 41 40 45 > 3 | 5034, 5034, 0, 0, 8 , 48, 0 = | 0 0 0 0 0 42 36 43 44 > 43 43 44 48 > * >=20 > 4) TRex patch issue solved >=20 > I40E_QINT_RQCTL_ITR_INDX_MASK > #define RTE_LIBRTE_I40E_ITR_INTERVAL -1 >=20 > latency : 8usec >=20 > -Latency stats enabled > Cpu Utilization : 0.1 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter= max > window > | , , , , average , max , (usec) > ------------------------------------------------------------------------= --------------------- > ------------------- > 0 | 9501, 9501, 0, 0, 7 , 21, 0 = | 21 9 9 12 9 9 14 12 8 9 > 9 9 8 > 1 | 9501, 9501, 0, 0, 7 , 25, 0 = | 22 8 9 12 9 9 15 12 8 8 > 9 9 8 > 2 | 9501, 9501, 0, 0, 6 , 26, 0 = | 22 9 10 9 10 10 15 13 8 > 9 9 9 8 > 3 | 9501, 9501, 0, 0, 6 , 32, 0 = | 22 8 9 9 9 9 15 12 8 7 > 9 9 9 >=20 > Thanks, > Hanoh >=20 > From: Wu, Jingjing [mailto:jingjing.wu@intel.com] > Sent: Tuesday, November 28, 2017 5:01 PM > To: Hanoch Haim (hhaim) > Cc: dev@dpdk.org > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttl= ing > setting in PF >=20 > Hi, Hanoch >=20 > If DPDK PF, the commit affects that because it introduces an argument > (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx wi= th > default value 32us. >=20 > If you'd like to get the descriptor write back immediately, you can set > "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=3D0" in config/common_base file. >=20 > Or you can just change the definition of > I40E_QUEUE_ITR_INTERVAL_DEFAULT like: > #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */ >=20 > Thanks > Jingjing >=20 > From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] > Sent: Tuesday, November 28, 2017 9:14 PM > To: Wu, Jingjing > > Cc: dev@dpdk.org; Hanoch Haim (hhaim) > > > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttl= ing > setting in PF >=20 > Hi Jingjing, >=20 > 1. The issue is with DPDK PF. >=20 > 2. The rate is high ~10gb, one DP core, one latency core. >=20 > 3. The fix is here >=20 > /* Bind all RX queues to allocated MSIX interrupt */ > for (i =3D 0; i < nb_queue; i++) { > val =3D (msix_vect << I40E_QINT_RQCTL_MSI= X_INDX_SHIFT) | > #ifdef TREX_PATCH > I40E_QINT_RQCTL_ITR_INDX_= MASK | > << low latency 11b =3D NoITR > #else > itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT= | > << high spkies > #endif >=20 > I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val); >=20 > The Interrupt Throttling ITR is configure using a different setting using= a > different register here : >=20 >=20 > 4. The ITR_INTERVAL is 32 usec and it affect a different PF register >=20 > #define I40E_ITR_INDEX_DEFAULT 0 > #define I40E_ITR_INDEX_NONE 3 > #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */ >=20 >=20 > 5. My question is why the VF configuration affects PF INT_INTERVAL ?= Can > I remove my patch and fix this latency issue in the different way? >=20 > Thanks, > Hanoh >=20 > From: Wu, Jingjing [mailto:jingjing.wu@intel.com] > Sent: Tuesday, November 28, 2017 2:50 PM > To: Hanoch Haim (hhaim); dev@dpdk.org > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttl= ing > setting in PF >=20 > Hi, Hanoch >=20 > Are you talking about i40 VF's latency? And you are using DPDK PF as host > driver? >=20 > In this case, we are setting the Interrupt Throttling (ITR) to be maximum= . > That is to say, if the packet rate is very slow , the receive descriptor = is written > back when ITR timeout, otherwise it is written back when cache line is f= ull (4 > descriptors/packets). I think that's why you saw the latency is varying. >=20 > If we change the ITR to minor, then huge number of interrupts will coming= to > core which impact performance. >=20 >=20 > Thanks > Jingjing >=20 > From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] > Sent: Friday, November 24, 2017 7:25 PM > To: dev@dpdk.org > Cc: Wu, Jingjing > > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttl= ing > setting in PF >=20 > Re-sending >=20 > Hanoh >=20 > From: Hanoch Haim (hhaim) > Sent: Monday, November 20, 2017 5:19 PM > To: dev@dpdk.org > Cc: Wu, Jingjing (jingjing.wu@intel.com); > Hanoch Haim (hhaim) > Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling = setting > in PF >=20 > Hi All, > While integrating dpdk17.11 into TRex latest code a new latency issue is > observed (i40e is very sensitive because it has very good resolution due= to > Qos configuration). > git bitsec found the following commit. > With this commit we observe high spikes of Rx latency (~40usec) vs (~8use= c). > Any idea why? > I can send how to reproduce this, it is very simple. >=20 > cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit > commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e > Author: Jingjing Wu > > Date: Thu Aug 24 09:57:51 2017 +0800 >=20 > net/i40e: fix interrupt throttling setting in PF >=20 > As no matter the PF host driver is DPDK or other kernel drivers, > they are sharing the same virtchnnl interfaces to communicate to VFs. > To follow the generic interface, DPDK PF need to set Interrupt > Throttling (ITR) index according to the rxitr_idx from virtchnnl > instead of ITR_NONE. >=20 > Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linu= x > VF") > Cc: stable@dpdk.org >=20 > Signed-off-by: Jingjing Wu > > >=20 >=20 >=20 > Thanks, > Hanoh