From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id AF4267CEF for ; Thu, 30 Nov 2017 18:46:59 +0100 (CET) Received: from fmsmga005.fm.intel.com ([10.253.24.32]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Nov 2017 09:46:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.45,341,1508828400"; d="scan'208,217";a="182086208" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by fmsmga005.fm.intel.com with ESMTP; 30 Nov 2017 09:46:58 -0800 Received: from fmsmsx154.amr.corp.intel.com (10.18.116.70) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 30 Nov 2017 09:46:57 -0800 Received: from shsmsx151.ccr.corp.intel.com (10.239.6.50) by FMSMSX154.amr.corp.intel.com (10.18.116.70) with Microsoft SMTP Server (TLS) id 14.3.319.2; Thu, 30 Nov 2017 09:46:57 -0800 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.213]) by SHSMSX151.ccr.corp.intel.com ([169.254.3.218]) with mapi id 14.03.0319.002; Fri, 1 Dec 2017 01:46:55 +0800 From: "Wu, Jingjing" To: "Hanoch Haim (hhaim)" CC: "dev@dpdk.org" Thread-Topic: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF Thread-Index: AdNiEt0UIw83Y/69THKQx+TvljuhYgDA+ARgAMu8aBAAANIg0AAD3UOgAGiGC4AAAeXjwA== Date: Thu, 30 Nov 2017 17:46:55 +0000 Message-ID: <9BB6961774997848B5B42BEC655768F810ECB645@SHSMSX103.ccr.corp.intel.com> References: <3f57eb6982af4bb9aae69bce67233d89@XCH-RTP-017.cisco.com> <9BB6961774997848B5B42BEC655768F810EC8AC3@SHSMSX103.ccr.corp.intel.com> <9BB6961774997848B5B42BEC655768F810EC8D87@SHSMSX103.ccr.corp.intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNzMxYzFiMWEtZTQ2NC00YTJjLWE0NTQtN2IwZmY4MGQyNmMzIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX0lDIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IlZiOXVVV0VNWUlNWmNQdExaZGE1Z3Z1K1NqNURHOVZvcVRQRHNrQzdqeTQ9In0= x-ctpclassification: CTP_IC dlp-product: dlpe-windows dlp-version: 11.0.0.116 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 17:47:01 -0000 Hi, Hanoch Thanks a lot for the trying. Are you using igb_uio to bind the device? I guess it would be because that fix is not complete, the overlap with Rx = interrupt mode is not considered. We will look into it. And it would be great if you can have a try on vfio_pci cases. Thanks Jingjing From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] Sent: Friday, December 1, 2017 12:48 AM To: Wu, Jingjing Cc: dev@dpdk.org; Hanoch Haim (hhaim) Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttlin= g setting in PF Hi Jingjing, I did that and see the results, It does not work as expected TRex command: $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c = 1 1) with the issue (without the patch) *itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | *#define RTE_LIBRTE_I40E_ITR_INTERVAL -1 latency : 40usec -Latency stats enabled Cpu Utilization : 0.2 % if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter = max window | , , , , average , max , (usec) --------------------------------------------------------------------------= -------------------------------------- 0 | 5603, 5603, 0, 0, 6 , 43, 1 = | 0 0 0 0 41 41 34 40 41 33 43 8 8 1 | 5603, 5603, 0, 0, 12 , 44, 24 = | 0 0 0 0 39 41 41 34 40 33 33 8 8 2 | 5603, 5603, 0, 0, 8 , 43, 5 = | 0 0 0 0 38 41 42 40 40 40 42 9 8 3 | 5603, 5603, 0, 0, 6 , 43, 1 = | 0 0 0 0 36 41 34 42 43 8 35 40 41 *** TRex is shutting down - cause: 'CTRL + C detected' 2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4 itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define RTE_LIBRTE_I40E_ITR_INTERVAL 4 latency : 40usec -Latency stats enabled Cpu Utilization : 0.2 % if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter = max window | , , , , average , max , (usec) --------------------------------------------------------------------------= -------------------------------------- 0 | 5034, 5034, 0, 0, 10 , 42, 0 = | 0 0 0 0 0 42 40 38 24 32 23 24 23 1 | 5034, 5034, 0, 0, 8 , 43, 0 = | 0 0 0 0 0 43 37 38 19 20 30 18 21 2 | 5034, 5034, 0, 0, 8 , 45, 0 = | 0 0 0 0 0 37 40 41 40 41 41 40 45 3 | 5034, 5034, 0, 0, 8 , 48, 0 = | 0 0 0 0 0 42 36 43 44 43 43 44 48 * 3) RTE_LIBRTE_I40E_ITR_INTERVAL 0 itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define RTE_LIBRTE_I40E_ITR_INTERVAL 0 latency : 40usec -Latency stats enabled Cpu Utilization : 0.2 % if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter = max window | , , , , average , max , (usec) --------------------------------------------------------------------------= -------------------------------------- 0 | 5034, 5034, 0, 0, 10 , 42, 0 = | 0 0 0 0 0 42 40 38 24 32 23 24 23 1 | 5034, 5034, 0, 0, 8 , 43, 0 = | 0 0 0 0 0 43 37 38 19 20 30 18 21 2 | 5034, 5034, 0, 0, 8 , 45, 0 = | 0 0 0 0 0 37 40 41 40 41 41 40 45 3 | 5034, 5034, 0, 0, 8 , 48, 0 = | 0 0 0 0 0 42 36 43 44 43 43 44 48 * 4) TRex patch issue solved I40E_QINT_RQCTL_ITR_INDX_MASK #define RTE_LIBRTE_I40E_ITR_INTERVAL -1 latency : 8usec -Latency stats enabled Cpu Utilization : 0.1 % if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter = max window | , , , , average , max , (usec) --------------------------------------------------------------------------= -------------------------------------- 0 | 9501, 9501, 0, 0, 7 , 21, 0 = | 21 9 9 12 9 9 14 12 8 9 9 9 8 1 | 9501, 9501, 0, 0, 7 , 25, 0 = | 22 8 9 12 9 9 15 12 8 8 9 9 8 2 | 9501, 9501, 0, 0, 6 , 26, 0 = | 22 9 10 9 10 10 15 13 8 9 9 9 8 3 | 9501, 9501, 0, 0, 6 , 32, 0 = | 22 8 9 9 9 9 15 12 8 7 9 9 9 Thanks, Hanoh From: Wu, Jingjing [mailto:jingjing.wu@intel.com] Sent: Tuesday, November 28, 2017 5:01 PM To: Hanoch Haim (hhaim) Cc: dev@dpdk.org Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttlin= g setting in PF Hi, Hanoch If DPDK PF, the commit affects that because it introduces an argument (itr_= idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with defa= ult value 32us. If you'd like to get the descriptor write back immediately, you can set "CO= NFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=3D0" in config/common_base file. Or you can just change the definition of I40E_QUEUE_ITR_INTERVAL_DEFAULT li= ke: #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */ Thanks Jingjing From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] Sent: Tuesday, November 28, 2017 9:14 PM To: Wu, Jingjing > Cc: dev@dpdk.org; Hanoch Haim (hhaim) > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttlin= g setting in PF Hi Jingjing, 1. The issue is with DPDK PF. 2. The rate is high ~10gb, one DP core, one latency core. 3. The fix is here /* Bind all RX queues to allocated MSIX interrupt */ for (i =3D 0; i < nb_queue; i++) { val =3D (msix_vect << I40E_QINT_RQCTL_MSIX_= INDX_SHIFT) | #ifdef TREX_PATCH I40E_QINT_RQCTL_ITR_INDX_MA= SK | << low latency 11b = =3D NoITR #else itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |= << high spkies #endif I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val); The Interrupt Throttling ITR is configure using a different setting using a= different register here : 4. The ITR_INTERVAL is 32 usec and it affect a different PF register #define I40E_ITR_INDEX_DEFAULT 0 #define I40E_ITR_INDEX_NONE 3 #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */ 5. My question is why the VF configuration affects PF INT_INTERVAL ? C= an I remove my patch and fix this latency issue in the different way? Thanks, Hanoh From: Wu, Jingjing [mailto:jingjing.wu@intel.com] Sent: Tuesday, November 28, 2017 2:50 PM To: Hanoch Haim (hhaim); dev@dpdk.org Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttlin= g setting in PF Hi, Hanoch Are you talking about i40 VF's latency? And you are using DPDK PF as host d= river? In this case, we are setting the Interrupt Throttling (ITR) to be maximum. = That is to say, if the packet rate is very slow , the receive descriptor is= written back when ITR timeout, otherwise it is written back when cache li= ne is full (4 descriptors/packets). I think that's why you saw the latency = is varying. If we change the ITR to minor, then huge number of interrupts will coming t= o core which impact performance. Thanks Jingjing From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com] Sent: Friday, November 24, 2017 7:25 PM To: dev@dpdk.org Cc: Wu, Jingjing > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttlin= g setting in PF Re-sending Hanoh From: Hanoch Haim (hhaim) Sent: Monday, November 20, 2017 5:19 PM To: dev@dpdk.org Cc: Wu, Jingjing (jingjing.wu@intel.com); Han= och Haim (hhaim) Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling se= tting in PF Hi All, While integrating dpdk17.11 into TRex latest code a new latency issue is ob= served (i40e is very sensitive because it has very good resolution due to = Qos configuration). git bitsec found the following commit. With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec)= . Any idea why? I can send how to reproduce this, it is very simple. cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e Author: Jingjing Wu > Date: Thu Aug 24 09:57:51 2017 +0800 net/i40e: fix interrupt throttling setting in PF As no matter the PF host driver is DPDK or other kernel drivers, they are sharing the same virtchnnl interfaces to communicate to VFs. To follow the generic interface, DPDK PF need to set Interrupt Throttling (ITR) index according to the rxitr_idx from virtchnnl instead of ITR_NONE. Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux = VF") Cc: stable@dpdk.org Signed-off-by: Jingjing Wu > Thanks, Hanoh