Re: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting in PF

["Xing, Beilei" <beilei.xing@intel.com>]

Hi,

I tested Rx latency with testpmd and ixia, found that the interval configuration works on X710.

With the default configuration, the latency is about 32us. 
When RTE_LIBRTE_I40E_ITR_INTERVAL=0, the max latency is < 8us.
When RTE_LIBRTE_I40E_ITR_INTERVAL=8160, the max latency is about 8ms.

My test steps:
1. connect a X710 port with ixia, and bind the port to igb_uio
2. run testpmd and start io forwding
3. send a burst (5 packets) with ixia to X710 port
4. check the latency on ixia.

Best Regards,
Beilei

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wu, Jingjing
> Sent: Friday, December 1, 2017 1:47 AM
> To: Hanoch Haim (hhaim) <hhaim@cisco.com>
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
> 
> Hi, Hanoch
> 
> Thanks a lot for the trying. Are you using igb_uio to bind the device?
> 
> I guess it would be because that fix is not complete, the overlap with  Rx
> interrupt mode is not considered. We will look into it.
> And it would be great if you can have a try on vfio_pci cases.
> 
> Thanks
> Jingjing
> 
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Friday, December 1, 2017 12:48 AM
> To: Wu, Jingjing <jingjing.wu@intel.com>
> Cc: dev@dpdk.org; Hanoch Haim (hhaim) <hhaim@cisco.com>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
> 
> Hi Jingjing,
> I did that and see the results, It does not work as expected
> 
> TRex command:
> $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c 1
> 
> 
> 
> 1) with the issue (without the patch)
> *itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | *#define
> RTE_LIBRTE_I40E_ITR_INTERVAL -1
> 
> 
> latency : 40usec
> 
> -Latency stats enabled
>  Cpu Utilization : 0.2 %
>  if|   tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max
> window
>    |         ,        ,          ,     ,   average   ,   max  ,    (usec)
>  ---------------------------------------------------------------------------------------------
> -------------------
>  0 |     5603,    5603,         0,    0,          6  ,      43,       1      |  0  0  0  0  41  41  34  40  41
> 33  43  8  8
>  1 |     5603,    5603,         0,    0,         12  ,      44,      24      |  0  0  0  0  39  41  41  34  40
> 33  33  8  8
>  2 |     5603,    5603,         0,    0,          8  ,      43,       5      |  0  0  0  0  38  41  42  40  40
> 40  42  9  8
>  3 |     5603,    5603,         0,    0,          6  ,      43,       1      |  0  0  0  0  36  41  34  42  43
> 8  35  40  41
>  *** TRex is shutting down - cause: 'CTRL + C detected'
> 
> 2) with  RTE_LIBRTE_I40E_ITR_INTERVAL 4
> 
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define
> RTE_LIBRTE_I40E_ITR_INTERVAL 4
> 
> latency : 40usec
> 
> -Latency stats enabled
>  Cpu Utilization : 0.2 %
>  if|   tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max
> window
>    |         ,        ,          ,     ,   average   ,   max  ,    (usec)
>  ---------------------------------------------------------------------------------------------
> -------------------
>  0 |     5034,    5034,         0,    0,         10  ,      42,       0      |  0  0  0  0  0  42  40  38  24
> 32  23  24  23
>  1 |     5034,    5034,         0,    0,          8  ,      43,       0      |  0  0  0  0  0  43  37  38  19
> 20  30  18  21
>  2 |     5034,    5034,         0,    0,          8  ,      45,       0      |  0  0  0  0  0  37  40  41  40
> 41  41  40  45
>  3 |     5034,    5034,         0,    0,          8  ,      48,       0      |  0  0  0  0  0  42  36  43  44
> 43  43  44  48
>  *
> 
> 
> 3) RTE_LIBRTE_I40E_ITR_INTERVAL 0
> 
> itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define
> RTE_LIBRTE_I40E_ITR_INTERVAL 0
> 
> latency : 40usec
> 
> -Latency stats enabled
>  Cpu Utilization : 0.2 %
>  if|   tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max
> window
>    |         ,        ,          ,     ,   average   ,   max  ,    (usec)
>  ---------------------------------------------------------------------------------------------
> -------------------
>  0 |     5034,    5034,         0,    0,         10  ,      42,       0      |  0  0  0  0  0  42  40  38  24
> 32  23  24  23
>  1 |     5034,    5034,         0,    0,          8  ,      43,       0      |  0  0  0  0  0  43  37  38  19
> 20  30  18  21
>  2 |     5034,    5034,         0,    0,          8  ,      45,       0      |  0  0  0  0  0  37  40  41  40
> 41  41  40  45
>  3 |     5034,    5034,         0,    0,          8  ,      48,       0      |  0  0  0  0  0  42  36  43  44
> 43  43  44  48
>  *
> 
> 4) TRex patch issue solved
> 
> I40E_QINT_RQCTL_ITR_INDX_MASK
> #define RTE_LIBRTE_I40E_ITR_INTERVAL -1
> 
> latency : 8usec
> 
> -Latency stats enabled
>  Cpu Utilization : 0.1 %
>  if|   tx_ok , rx_ok  , rx check ,error,       latency (usec) ,    Jitter          max
> window
>    |         ,        ,          ,     ,   average   ,   max  ,    (usec)
>  ---------------------------------------------------------------------------------------------
> -------------------
>  0 |     9501,    9501,         0,    0,          7  ,      21,       0      |  21  9  9  12  9  9  14  12  8  9
> 9  9  8
>  1 |     9501,    9501,         0,    0,          7  ,      25,       0      |  22  8  9  12  9  9  15  12  8  8
> 9  9  8
>  2 |     9501,    9501,         0,    0,          6  ,      26,       0      |  22  9  10  9  10  10  15  13  8
> 9  9  9  8
>  3 |     9501,    9501,         0,    0,          6  ,      32,       0      |  22  8  9  9  9  9  15  12  8  7
> 9  9  9
> 
> Thanks,
> Hanoh
> 
> From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
> Sent: Tuesday, November 28, 2017 5:01 PM
> To: Hanoch Haim (hhaim)
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
> 
> Hi, Hanoch
> 
> If DPDK PF, the commit affects that because it introduces an argument
> (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with
> default value 32us.
> 
> If you'd like to get the descriptor write back immediately, you can set
> "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file.
> 
> Or you can just change the definition of
> I40E_QUEUE_ITR_INTERVAL_DEFAULT like:
> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */
> 
> Thanks
> Jingjing
> 
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Tuesday, November 28, 2017 9:14 PM
> To: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim)
> <hhaim@cisco.com<mailto:hhaim@cisco.com>>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
> 
> Hi Jingjing,
> 
> 1.      The issue is with DPDK PF.
> 
> 2.      The rate is high ~10gb, one DP core, one latency core.
> 
> 3.      The fix is here
> 
> /* Bind all RX queues to allocated MSIX interrupt */
>                 for (i = 0; i < nb_queue; i++) {
>                                 val = (msix_vect << I40E_QINT_RQCTL_MSIX_INDX_SHIFT) |
>             #ifdef TREX_PATCH
>                                                 I40E_QINT_RQCTL_ITR_INDX_MASK |
> << low latency 11b = NoITR
>             #else
>                                 itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT |
> << high spkies
>             #endif
> 
>              I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val);
> 
> The Interrupt Throttling ITR is configure using a different setting using a
> different register here :
> 
> 
> 4.      The ITR_INTERVAL is 32 usec and it affect a different PF register
> 
> #define I40E_ITR_INDEX_DEFAULT          0
> #define I40E_ITR_INDEX_NONE             3
> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */
> 
> 
> 5.      My question is why the VF configuration affects PF INT_INTERVAL ? Can
> I remove my patch and fix this latency issue in the different way?
> 
> Thanks,
> Hanoh
> 
> From: Wu, Jingjing [mailto:jingjing.wu@intel.com]
> Sent: Tuesday, November 28, 2017 2:50 PM
> To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
> 
> Hi, Hanoch
> 
> Are you talking about i40 VF's latency? And you are using DPDK PF as host
> driver?
> 
> In this case, we are setting the Interrupt Throttling (ITR) to be maximum.
> That is to say, if the packet rate is very slow , the receive descriptor is written
> back  when ITR timeout, otherwise it is written back when cache line is full (4
> descriptors/packets). I think that's why you saw the latency is varying.
> 
> If we change the ITR to minor, then huge number of interrupts will coming to
> core which impact performance.
> 
> 
> Thanks
> Jingjing
> 
> From: Hanoch Haim (hhaim) [mailto:hhaim@cisco.com]
> Sent: Friday, November 24, 2017 7:25 PM
> To: dev@dpdk.org<mailto:dev@dpdk.org>
> Cc: Wu, Jingjing <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling
> setting in PF
> 
> Re-sending
> 
> Hanoh
> 
> From: Hanoch Haim (hhaim)
> Sent: Monday, November 20, 2017 5:19 PM
> To: dev@dpdk.org<mailto:dev@dpdk.org>
> Cc: Wu, Jingjing (jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>);
> Hanoch Haim (hhaim)
> Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling setting
> in PF
> 
> Hi All,
> While integrating dpdk17.11 into TRex latest code a new latency issue is
> observed  (i40e is very sensitive because it has very good resolution due to
> Qos configuration).
> git bitsec found the following commit.
> With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec).
> Any idea why?
> I can send how to reproduce this, it is very simple.
> 
> cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit
> commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e
> Author: Jingjing Wu <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> Date:   Thu Aug 24 09:57:51 2017 +0800
> 
>     net/i40e: fix interrupt throttling setting in PF
> 
>     As no matter the PF host driver is DPDK or other kernel drivers,
>     they are sharing the same virtchnnl interfaces to communicate to VFs.
>     To follow the generic interface, DPDK PF need to set Interrupt
>     Throttling (ITR) index according to the rxitr_idx from virtchnnl
>     instead of ITR_NONE.
> 
>     Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux
> VF")
>     Cc: stable@dpdk.org<mailto:stable@dpdk.org>
> 
>     Signed-off-by: Jingjing Wu
> <jingjing.wu@intel.com<mailto:jingjing.wu@intel.com>>
> 
> 
> 
> Thanks,
> Hanoh