From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by dpdk.org (Postfix) with ESMTP id 87CF21B454 for ; Tue, 26 Jun 2018 11:15:31 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 26 Jun 2018 02:15:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.51,274,1526367600"; d="scan'208";a="61352680" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.83]) ([10.237.221.83]) by fmsmga002.fm.intel.com with ESMTP; 26 Jun 2018 02:15:28 -0700 To: "Zhang, Qi Z" , Stephen Hemminger Cc: "Xing, Beilei" , "Wu, Jingjing" , "Yu, De" , "dev@dpdk.org" References: <20180622004414.28849-1-qi.z.zhang@intel.com> <20180622084428.61154907@xeon-e3> <039ED4275CED7440929022BC67E706115323C1AC@SHSMSX103.ccr.corp.intel.com> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= xsFNBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABzSVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+wsF+BBMBAgAoAhsDBgsJCAcDAgYVCAIJCgsE FgIDAQIeAQIXgAUCWZR3VQUJB33WBQAKCRD5M+tD3xNhH6DWEACVhEb8q1epPwZrUDoxzu7E TS1b8tmabOmnjXZRs6+EXgUVHkp2xxkCfDmL3pa5bC0G/74aJnWjNsdvE05V1cb4YK4kRQ62 FwDQ+hlrFrwFB3PtDZk1tpkzCRHvJgnIil+0MuEh32Y57ig6hy8yO8ql7Lohyrnpfk/nNpm4 jQGEF5qEeHcEFe1AZQlPHN/STno8NZSz2nl0b2cw+cujN1krmvB52Ah/2KugQ6pprVyrGrzB c34ZQO9OsmSjJlETCZk6EZzuhfe16iqBFbOSadi9sPcJRwaUQBid+xdFWl7GQ8qC3zNPibSF HmU43yBZUqJDZlhIcl6/cFpOSjv2sDWdtjEXTDn5y/0FsuY0mFE78ItC4kCTIVk17VZoywcd fmbbnwOSWzDq7hiUYuQGkIudJw5k/A1CMsyLkoUEGN3sLfsw6KASgS4XrrmPO4UVr3mH5bP1 yC7i1OVNpzvOxtahmzm481ID8sk72GC2RktTOHb0cX+qdoiMMfYgo3wRRDYCBt6YoGYUxF1p msjocXyqToKhhnFbXLaZlVfnQ9i2i8jsj9SKig+ewC2p3lkPj6ncye9q95bzhmUeJO6sFhJg Hiz6syOMg8yCcq60j07airybAuHIDNFWk0gaWAmtHZxLObZx2PVn2nv9kLYGohFekw0AOsIW ta++5m48dnCoAc7BTQRX1ky+ARAApzQNvXvE2q1LAS+Z+ni2R13Bb1cDS1ZYq1jgpR13+OKN ipzd8MPngRJilXxBaPTErhgzR0vGcNTYhjGMSyFIHVOoBq1VbP1a0Fi/NqWzJOowo/fDfgVy K4vuitc/gCJs+2se4hdZA4EQJxVlNM51lgYDNpjPGIA43MX15OLAip73+ho6NPBMuc5qse3X pAClNhBKfENRCWN428pi3WVkT+ABRTE0taxjJNP7bb+9TQYNRqGwnGzX5/XISv44asWIQCaq vOkXSUJLd//cdVNTqtL1wreCVVR5pMXj7VIrlk07fmmJVALCmGbFr53BMb8O+8dgK2A5mitM n44d+8KdJWOwziRxcaMk/LclmZS3Iv1TERtiWt98Y9AjeAtcgYPkA3ld0BcUKONogP8pHVz1 Ed3s5rDQ91yr1S0wuAzW91fxGUO4wY+uPmxCtFVuBgd9VT9NAKTUL0qHM7CDgCnZPe0TW6Zj 8OqtdCCyAfvU9cW5xWM7Icxhde6AtPxhDSBwE8fL2ZmrDmaA4jmUKXp3i4JxRPSX84S08b+s DWXHPxy10UFU5A7EK/BEbZAKBwn9ROfm+WK+6X5xOGLoRE++OqNuUudxC1GDyLOPaqCbBCS9 +P6HsTHzxsjyJa27n4jcrcuY3P9TEcFJYSZSeSDh8mVGvugi0exnSJrrBZDyVCcAEQEAAcLB ZQQYAQIADwIbDAUCWZR1ZwUJA59cIQAKCRD5M+tD3xNhH5b+D/9XG44Ci6STdcA5RO/ur05J EE3Ux1DCHZ5V7vNAtX/8Wg4l4GZfweauXwuJ1w7Sp7fklwcNC6wsceI+EmNjGMqfIaukGetG +jBGqsQ7moOZodfXUoCK98gblKgt/BPYMVidzlGC8Q/+lZg1+o29sPnwImW+MXt/Z5az/Z17 Qc265g+p5cqJHzq6bpQdnF7Fu6btKU/kv6wJghENvgMXBuyThqsyFReJWFh2wfaKyuix3Zyj ccq7/blkhzIKmtFWgDcgaSc2UAuJU+x9nuYjihW6WobpKP/nlUDu3BIsbIq09UEke+uE/QK+ FJ8PTJkAsXOf1Bc2C0XbW4Y2hf103+YY6L8weUCBsWC5VH5VtVmeuh26ENURclwfeXhWQ9Og 77yzpTXWr5g1Z0oLpYpWPv745J4bE7pv+dzxOrFdM1xNkzY2pvXph/A8OjxZNQklDkHQ7PIB Lki5L2F4XkEOddUUQchJwzMqTPsggPDmGjgLZrqgO+s4ECZK5+nLD3HEpAbPa3JLDaScy+90 Nu1lAqPUHSnP3vYZVw85ZYm6UCxHE4VLMnnJsN09ZhsOSVR+GyP5Nyw9rT1V3lcsuH7M5Naa 2Xobn9m7l9bRCD/Ji8kG15eV1WTxx1HXVQGjdUYDI7UwegBNbwMLh17XDy+3sn/6SgcqtECA Q6pZKA2mTQxEKMLBZQQYAQIADwIbDAUCWZR3hQUJA59eRwAKCRD5M+tD3xNhH4a/D/4jLAZu UhvU1swWcNEVVCELZ0D3LOV14XcY2MXa3QOpeZ9Bgq7YYJ4S5YXK+SBQS0FkRZdjGNvlGZoG ZdpU+NsQmQFhqHGwX0IT9MeTFM8uvKgxNKGwMVcV9g0IOqwBhGHne+BFboRA9362fgGW5AYQ zT0mzzRKEoOh4r3AQvbM6kLISxo0k1ujdYiI5nj/5WoKDqxTwwfuN1uDUHsWo3tzenRmpMyU NyW3Dc+1ajvXLyo09sRRq7BnM99Rix1EGL8Qhwy+j0YAv+FuspWxUX9FxXYho5PvGLHLsHfK FYQ7x/RRbpMjkJWVfIe/xVnfvn4kz+MTA5yhvsuNi678fLwY9hBP0y4lO8Ob2IhEPdfnTuIs tFVxXuelJ9xAe5TyqP0f+fQjf1ixsBZkqOohsBXDfje0iaUpYa/OQ/BBeej0dUdg2JEu4jAC x41HpVCnP9ipLpD0fYz1d/dX0F/VY2ovW6Eba/y/ngOSAR6C+u881m7oH2l0G47MTwkaQCBA bLGXPj4TCdX3lftqt4bcBPBJ+rFAnJmRHtUuyyaewBnZ81ZU2YAptqFM1kTh+aSvMvGhfVsQ qZL2rk2OPN1hg+KXhErlbTZ6oPtLCFhSHQmuxQ4oc4U147wBTUuOdwNjtnNatUhRCp8POc+3 XphVR5G70mnca1E2vzC77z+XSlTyRA== Message-ID: <6744a9e1-8770-31a0-89b0-a0c0ff74f05a@intel.com> Date: Tue, 26 Jun 2018 10:15:28 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <039ED4275CED7440929022BC67E706115323C1AC@SHSMSX103.ccr.corp.intel.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [PATCH v2] net/i40e: remove VF interrupt handler X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jun 2018 09:15:32 -0000 On 6/24/2018 11:56 AM, Zhang, Qi Z wrote: > Hi Stephen: > >> -----Original Message----- >> From: Stephen Hemminger [mailto:stephen@networkplumber.org] >> Sent: Friday, June 22, 2018 11:44 PM >> To: Zhang, Qi Z >> Cc: Xing, Beilei ; Wu, Jingjing ; >> Yu, De ; dev@dpdk.org >> Subject: Re: [dpdk-dev] [PATCH v2] net/i40e: remove VF interrupt handler >> >> On Fri, 22 Jun 2018 08:44:14 +0800 >> Qi Zhang wrote: >> >>> For i40evf, internal rx interrupt and adminq interrupt share the same >>> source, that cause a lot cpu cycles be wasted on interrupt handler on >>> rx path. This is complained by customers which require low latency >>> (when set I40E_ITR_INTERVAL to small value), but have to be sufferred >>> by tremendous interrupts handling that eat significant CPU resources. >>> >>> The patch disable pci interrupt and remove the interrupt handler, >>> replace it with a low frequency (50ms) interrupt polling daemon which >>> is implemented by registering a alarm callback periodly, this save CPU >>> time significently: On a typical x86 server with 2.1GHz CPU, with low >>> latency configure (32us) we saw CPU usage from top commmand reduced >>> from 20% to 0% on management core in testpmd). >>> >>> Also with the new method we can remove compile option: >>> I40E_ITR_INTERVAL which is used to balance between low latency and low >> CPU usage previously. >>> Now we don't need it since we can reach both at same time. >>> >>> Suggested-by: Jingjing Wu >>> Signed-off-by: Qi Zhang >>> --- >>> >>> v2: >>> - update doc >>> >>> config/common_base | 2 -- >>> doc/guides/nics/i40e.rst | 5 ----- >>> drivers/net/i40e/i40e_ethdev.c | 3 +-- >>> drivers/net/i40e/i40e_ethdev.h | 22 +++++++++++----------- >>> drivers/net/i40e/i40e_ethdev_vf.c | 36 >>> ++++++++++++++---------------------- >>> 5 files changed, 26 insertions(+), 42 deletions(-) >>> >>> diff --git a/config/common_base b/config/common_base index >>> 6b0d1cbbb..9e21c6865 100644 >>> --- a/config/common_base >>> +++ b/config/common_base >>> @@ -264,8 +264,6 @@ CONFIG_RTE_LIBRTE_I40E_INC_VECTOR=y >>> CONFIG_RTE_LIBRTE_I40E_16BYTE_RX_DESC=n >>> CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_PF=64 >>> CONFIG_RTE_LIBRTE_I40E_QUEUE_NUM_PER_VM=4 >>> -# interval up to 8160 us, aligned to 2 (or default value) >>> -CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=-1 >>> >>> # >>> # Compile burst-oriented FM10K PMD >>> diff --git a/doc/guides/nics/i40e.rst b/doc/guides/nics/i40e.rst index >>> 18549bf5a..3fc4ceac7 100644 >>> --- a/doc/guides/nics/i40e.rst >>> +++ b/doc/guides/nics/i40e.rst >>> @@ -96,11 +96,6 @@ Please note that enabling debugging options may >> affect system performance. >>> >>> Number of queues reserved for each VMDQ Pool. >>> >>> -- ``CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL`` (default ``-1``) >>> - >>> - Interrupt Throttling interval. >>> - >>> - >>> Runtime Config Options >>> ~~~~~~~~~~~~~~~~~~~~~~ >>> >>> diff --git a/drivers/net/i40e/i40e_ethdev.c >>> b/drivers/net/i40e/i40e_ethdev.c index 13c5d3296..c8f9566e0 100644 >>> --- a/drivers/net/i40e/i40e_ethdev.c >>> +++ b/drivers/net/i40e/i40e_ethdev.c >>> @@ -1829,8 +1829,7 @@ __vsi_queues_bind_intr(struct i40e_vsi *vsi, >> uint16_t msix_vect, >>> /* Write first RX queue to Link list register as the head element */ >>> if (vsi->type != I40E_VSI_SRIOV) { >>> uint16_t interval = >>> - i40e_calc_itr_interval(RTE_LIBRTE_I40E_ITR_INTERVAL, 1, >>> - pf->support_multi_driver); >>> + i40e_calc_itr_interval(1, pf->support_multi_driver); >>> >>> if (msix_vect == I40E_MISC_VEC_ID) { >>> I40E_WRITE_REG(hw, I40E_PFINT_LNKLST0, diff --git >>> a/drivers/net/i40e/i40e_ethdev.h b/drivers/net/i40e/i40e_ethdev.h >>> index 11c4c76bd..599993dac 100644 >>> --- a/drivers/net/i40e/i40e_ethdev.h >>> +++ b/drivers/net/i40e/i40e_ethdev.h >>> @@ -178,7 +178,7 @@ enum i40e_flxpld_layer_idx { >>> #define I40E_ITR_INDEX_NONE 3 >>> #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */ >>> #define I40E_QUEUE_ITR_INTERVAL_MAX 8160 /* 8160 us */ >>> -#define I40E_VF_QUEUE_ITR_INTERVAL_DEFAULT 8160 /* 8160 us */ >>> +#define I40E_VF_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */ >>> /* Special FW support this floating VEB feature */ #define >>> FLOATING_VEB_SUPPORTED_FW_MAJ 5 #define >> FLOATING_VEB_SUPPORTED_FW_MIN >>> 0 @@ -1328,17 +1328,17 @@ i40e_align_floor(int n) } >>> >>> static inline uint16_t >>> -i40e_calc_itr_interval(int16_t interval, bool is_pf, bool >>> is_multi_drv) >>> +i40e_calc_itr_interval(bool is_pf, bool is_multi_drv) >>> { >>> - if (interval < 0 || interval > I40E_QUEUE_ITR_INTERVAL_MAX) { >>> - if (is_multi_drv) { >>> - interval = I40E_QUEUE_ITR_INTERVAL_MAX; >>> - } else { >>> - if (is_pf) >>> - interval = I40E_QUEUE_ITR_INTERVAL_DEFAULT; >>> - else >>> - interval = I40E_VF_QUEUE_ITR_INTERVAL_DEFAULT; >>> - } >>> + uint16_t interval = 0; >>> + >>> + if (is_multi_drv) { >>> + interval = I40E_QUEUE_ITR_INTERVAL_MAX; >>> + } else { >>> + if (is_pf) >>> + interval = I40E_QUEUE_ITR_INTERVAL_DEFAULT; >>> + else >>> + interval = I40E_VF_QUEUE_ITR_INTERVAL_DEFAULT; >>> } >>> >>> /* Convert to hardware count, as writing each 1 represents 2 us */ >>> diff --git a/drivers/net/i40e/i40e_ethdev_vf.c >>> b/drivers/net/i40e/i40e_ethdev_vf.c >>> index 804e44530..ad5c069e8 100644 >>> --- a/drivers/net/i40e/i40e_ethdev_vf.c >>> +++ b/drivers/net/i40e/i40e_ethdev_vf.c >>> @@ -44,6 +44,8 @@ >>> #define I40EVF_BUSY_WAIT_COUNT 50 >>> #define MAX_RESET_WAIT_CNT 20 >>> >>> +#define I40EVF_ALARM_INTERVAL 50000 /* us */ >>> + >>> struct i40evf_arq_msg_info { >>> enum virtchnl_ops ops; >>> enum i40e_status_code result; >>> @@ -1133,7 +1135,7 @@ i40evf_init_vf(struct rte_eth_dev *dev) >>> struct i40e_hw *hw = >> I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private); >>> struct i40e_vf *vf = >> I40EVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); >>> uint16_t interval = >>> - i40e_calc_itr_interval(RTE_LIBRTE_I40E_ITR_INTERVAL, 0, 0); >>> + i40e_calc_itr_interval(0, 0); >>> >>> vf->adapter = I40E_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); >>> vf->dev_data = dev->data; >>> @@ -1370,7 +1372,7 @@ i40evf_handle_aq_msg(struct rte_eth_dev *dev) >>> * void >>> */ >>> static void >>> -i40evf_dev_interrupt_handler(void *param) >>> +i40evf_dev_alarm_handler(void *param) >>> { >>> struct rte_eth_dev *dev = (struct rte_eth_dev *)param; >>> struct i40e_hw *hw = >> I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private); >>> @@ -1399,6 +1401,8 @@ i40evf_dev_interrupt_handler(void *param) >>> >>> done: >>> i40evf_enable_irq0(hw); >>> + rte_eal_alarm_set(I40EVF_ALARM_INTERVAL, >>> + i40evf_dev_alarm_handler, dev); >>> } >>> >>> static int >>> @@ -1442,12 +1446,8 @@ i40evf_dev_init(struct rte_eth_dev *eth_dev) >>> return -1; >>> } >>> >>> - /* register callback func to eal lib */ >>> - rte_intr_callback_register(&pci_dev->intr_handle, >>> - i40evf_dev_interrupt_handler, (void *)eth_dev); >>> - >>> - /* enable uio intr after callback register */ >>> - rte_intr_enable(&pci_dev->intr_handle); >>> + rte_eal_alarm_set(I40EVF_ALARM_INTERVAL, >>> + i40evf_dev_alarm_handler, eth_dev); >>> >>> /* configure and enable device interrupt */ >>> i40evf_enable_irq0(hw); >>> @@ -1836,7 +1836,7 @@ i40evf_dev_rx_queue_intr_enable(struct >> rte_eth_dev *dev, uint16_t queue_id) >>> struct rte_intr_handle *intr_handle = &pci_dev->intr_handle; >>> struct i40e_hw *hw = >> I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private); >>> uint16_t interval = >>> - i40e_calc_itr_interval(RTE_LIBRTE_I40E_ITR_INTERVAL, 0, 0); >>> + i40e_calc_itr_interval(0, 0); >>> uint16_t msix_intr; >>> >>> msix_intr = intr_handle->intr_vec[queue_id]; @@ -1859,8 +1859,6 @@ >>> i40evf_dev_rx_queue_intr_enable(struct rte_eth_dev *dev, uint16_t >>> queue_id) >>> >>> I40EVF_WRITE_FLUSH(hw); >>> >>> - rte_intr_enable(&pci_dev->intr_handle); >>> - >>> return 0; >>> } >>> >>> @@ -2023,10 +2021,8 @@ i40evf_dev_start(struct rte_eth_dev *dev) >>> * queue interrupt to other VFIO vectors. >>> * So clear uio/vfio intr/evevnfd first to avoid failure. >>> */ >>> - if (dev->data->dev_conf.intr_conf.rxq != 0) { >>> - rte_intr_disable(intr_handle); >>> + if (dev->data->dev_conf.intr_conf.rxq != 0) >>> rte_intr_enable(intr_handle); >>> - } >>> >>> i40evf_enable_queues_intr(dev); >>> >>> @@ -2050,6 +2046,9 @@ i40evf_dev_stop(struct rte_eth_dev *dev) >>> >>> PMD_INIT_FUNC_TRACE(); >>> >>> + if (dev->data->dev_conf.intr_conf.rxq != 0) >>> + rte_intr_disable(intr_handle); >>> + >>> if (hw->adapter_stopped == 1) >>> return; >>> i40evf_stop_queues(dev); >>> @@ -2285,9 +2284,8 @@ static void >>> i40evf_dev_close(struct rte_eth_dev *dev) { >>> struct i40e_hw *hw = >> I40E_DEV_PRIVATE_TO_HW(dev->data->dev_private); >>> - struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev); >>> - struct rte_intr_handle *intr_handle = &pci_dev->intr_handle; >>> >>> + rte_eal_alarm_cancel(i40evf_dev_alarm_handler, dev); >>> i40evf_dev_stop(dev); >>> i40e_dev_free_queues(dev); >>> /* >>> @@ -2300,12 +2298,6 @@ i40evf_dev_close(struct rte_eth_dev *dev) >>> >>> i40evf_reset_vf(hw); >>> i40e_shutdown_adminq(hw); >>> - /* disable uio intr before callback unregister */ >>> - rte_intr_disable(intr_handle); >>> - >>> - /* unregister callback func from eal lib */ >>> - rte_intr_callback_unregister(intr_handle, >>> - i40evf_dev_interrupt_handler, dev); >>> i40evf_disable_irq0(hw); >>> } >>> >> >> Rather than adding a polling routine internally, why not change the driver to >> not support Link State or receive interrupts. Better yet, let the application >> decide. >> Keep the interrupt logic but only enable interrupts if application has requested >> LSC or recveive interrupt mode. > > The interrupt handler is not only for LSC (actually VF does not support LSC) or rx interrupt mode, > it is used for PF to VF message through admin queue which is always required. I guess the question is, is it possible to disable Rx interrupts? And if possible can user control this enable/disable per interrupt source?