From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id EDD0F8E67 for ; Tue, 22 Dec 2015 08:28:13 +0100 (CET) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga103.fm.intel.com with ESMTP; 21 Dec 2015 23:28:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.20,463,1444719600"; d="scan'208";a="867073559" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by fmsmga001.fm.intel.com with ESMTP; 21 Dec 2015 23:28:12 -0800 Received: from FMSMSX110.amr.corp.intel.com (10.18.116.10) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.248.2; Mon, 21 Dec 2015 23:28:11 -0800 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by fmsmsx110.amr.corp.intel.com (10.18.116.10) with Microsoft SMTP Server (TLS) id 14.3.248.2; Mon, 21 Dec 2015 23:28:11 -0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.190]) by shsmsx102.ccr.corp.intel.com ([169.254.2.158]) with mapi id 14.03.0248.002; Tue, 22 Dec 2015 15:28:09 +0800 From: "Qiu, Michael" To: "He, Shaopeng" , "dev@dpdk.org" Thread-Topic: [dpdk-dev] [PATCH v4 2/6] fm10k: setup rx queue interrupts for PF and VF Thread-Index: AQHRO9lJL0dcEeJYzkGaHiOLODKKdA== Date: Tue, 22 Dec 2015 07:28:09 +0000 Message-ID: <533710CFB86FA344BFBF2D6802E6028622EFD1CC@SHSMSX101.ccr.corp.intel.com> References: <1446700329-10048-1-git-send-email-shaopeng.he@intel.com> <1450693192-14500-1-git-send-email-shaopeng.he@intel.com> <1450693192-14500-3-git-send-email-shaopeng.he@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [PATCH v4 2/6] fm10k: setup rx queue interrupts for PF and VF X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2015 07:28:14 -0000 On 12/21/2015 6:20 PM, Shaopeng He wrote:=0A= > In interrupt mode, each rx queue can have one interrupt to notify the up= =0A= > layer application when packets are available in that queue. Some queues= =0A= > also can share one interrupt.=0A= > Currently, fm10k needs one separate interrupt for mailbox. So, only those= =0A= > drivers which support multiple interrupt vectors e.g. vfio-pci can work= =0A= > in fm10k interrupt mode.=0A= > This patch uses the RXINT/INT_MAP registers to map interrupt causes=0A= > (rx queue and other events) to vectors, and enable these interrupts=0A= > through kernel drivers like vfio-pci.=0A= >=0A= > Signed-off-by: Shaopeng He =0A= > Acked-by: Jing Chen =0A= > ---=0A= > doc/guides/rel_notes/release_2_3.rst | 2 +=0A= > drivers/net/fm10k/fm10k.h | 3 ++=0A= > drivers/net/fm10k/fm10k_ethdev.c | 101 +++++++++++++++++++++++++++++= ++----=0A= > 3 files changed, 95 insertions(+), 11 deletions(-)=0A= >=0A= > diff --git a/doc/guides/rel_notes/release_2_3.rst b/doc/guides/rel_notes/= release_2_3.rst=0A= > index 99de186..2cb5ebd 100644=0A= > --- a/doc/guides/rel_notes/release_2_3.rst=0A= > +++ b/doc/guides/rel_notes/release_2_3.rst=0A= > @@ -4,6 +4,8 @@ DPDK Release 2.3=0A= > New Features=0A= > ------------=0A= > =0A= > +* **Added fm10k Rx interrupt support.**=0A= > +=0A= > =0A= > Resolved Issues=0A= > ---------------=0A= > diff --git a/drivers/net/fm10k/fm10k.h b/drivers/net/fm10k/fm10k.h=0A= > index e2f677a..770d6ba 100644=0A= > --- a/drivers/net/fm10k/fm10k.h=0A= > +++ b/drivers/net/fm10k/fm10k.h=0A= > @@ -129,6 +129,9 @@=0A= > #define RTE_FM10K_TX_MAX_FREE_BUF_SZ 64=0A= > #define RTE_FM10K_DESCS_PER_LOOP 4=0A= > =0A= > +#define FM10K_MISC_VEC_ID RTE_INTR_VEC_ZERO_OFFSET=0A= > +#define FM10K_RX_VEC_START RTE_INTR_VEC_RXTX_OFFSET=0A= > +=0A= > #define FM10K_SIMPLE_TX_FLAG ((uint32_t)ETH_TXQ_FLAGS_NOMULTSEGS | \=0A= > ETH_TXQ_FLAGS_NOOFFLOADS)=0A= > =0A= > diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_e= thdev.c=0A= > index d39c33b..a34c5e2 100644=0A= > --- a/drivers/net/fm10k/fm10k_ethdev.c=0A= > +++ b/drivers/net/fm10k/fm10k_ethdev.c=0A= > @@ -54,6 +54,8 @@=0A= > /* Number of chars per uint32 type */=0A= > #define CHARS_PER_UINT32 (sizeof(uint32_t))=0A= > #define BIT_MASK_PER_UINT32 ((1 << CHARS_PER_UINT32) - 1)=0A= > +/* default 1:1 map from queue ID to interrupt vector ID */=0A= > +#define Q2V(dev, queue_id) (dev->pci_dev->intr_handle.intr_vec[queue_id]= )=0A= > =0A= > static void fm10k_close_mbx_service(struct fm10k_hw *hw);=0A= > static void fm10k_dev_promiscuous_enable(struct rte_eth_dev *dev);=0A= > @@ -109,6 +111,8 @@ struct fm10k_xstats_name_off fm10k_hw_stats_tx_q_stri= ngs[] =3D {=0A= > =0A= > #define FM10K_NB_XSTATS (FM10K_NB_HW_XSTATS + FM10K_MAX_QUEUES_PF * \=0A= > (FM10K_NB_RX_Q_XSTATS + FM10K_NB_TX_Q_XSTATS))=0A= > +static int=0A= > +fm10k_dev_rxq_interrupt_setup(struct rte_eth_dev *dev);=0A= > =0A= > static void=0A= > fm10k_mbx_initlock(struct fm10k_hw *hw)=0A= > @@ -687,6 +691,7 @@ static int=0A= > fm10k_dev_rx_init(struct rte_eth_dev *dev)=0A= > {=0A= > struct fm10k_hw *hw =3D FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private)= ;=0A= > + struct rte_intr_handle *intr_handle =3D &dev->pci_dev->intr_handle;=0A= > int i, ret;=0A= > struct fm10k_rx_queue *rxq;=0A= > uint64_t base_addr;=0A= > @@ -694,10 +699,23 @@ fm10k_dev_rx_init(struct rte_eth_dev *dev)=0A= > uint32_t rxdctl =3D FM10K_RXDCTL_WRITE_BACK_MIN_DELAY;=0A= > uint16_t buf_size;=0A= > =0A= > - /* Disable RXINT to avoid possible interrupt */=0A= > - for (i =3D 0; i < hw->mac.max_queues; i++)=0A= > + /* enable RXINT for interrupt mode */=0A= > + i =3D 0;=0A= > + if (rte_intr_dp_is_en(intr_handle)) {=0A= > + for (; i < dev->data->nb_rx_queues; i++) {=0A= > + FM10K_WRITE_REG(hw, FM10K_RXINT(i), Q2V(dev, i));=0A= > + if (hw->mac.type =3D=3D fm10k_mac_pf)=0A= > + FM10K_WRITE_REG(hw, FM10K_ITR(Q2V(dev, i)),=0A= > + FM10K_ITR_AUTOMASK | FM10K_ITR_MASK_CLEAR);=0A= > + else=0A= > + FM10K_WRITE_REG(hw, FM10K_VFITR(Q2V(dev, i)),=0A= > + FM10K_ITR_AUTOMASK | FM10K_ITR_MASK_CLEAR);=0A= > + }=0A= > + }=0A= > + /* Disable other RXINT to avoid possible interrupt */=0A= > + for (; i < hw->mac.max_queues; i++)=0A= > FM10K_WRITE_REG(hw, FM10K_RXINT(i),=0A= > - 3 << FM10K_RXINT_TIMER_SHIFT);=0A= > + 3 << FM10K_RXINT_TIMER_SHIFT);=0A= > =0A= > /* Setup RX queues */=0A= > for (i =3D 0; i < dev->data->nb_rx_queues; ++i) {=0A= > @@ -1053,6 +1071,9 @@ fm10k_dev_start(struct rte_eth_dev *dev)=0A= > return diag;=0A= > }=0A= > =0A= > + if (fm10k_dev_rxq_interrupt_setup(dev))=0A= > + return -EIO;=0A= > +=0A= > diag =3D fm10k_dev_rx_init(dev);=0A= > if (diag) {=0A= > PMD_INIT_LOG(ERR, "RX init failed: %d", diag);=0A= > @@ -2072,7 +2093,7 @@ fm10k_dev_enable_intr_pf(struct rte_eth_dev *dev)= =0A= > uint32_t int_map =3D FM10K_INT_MAP_IMMEDIATE;=0A= > =0A= > /* Bind all local non-queue interrupt to vector 0 */=0A= > - int_map |=3D 0;=0A= > + int_map |=3D FM10K_MISC_VEC_ID;=0A= > =0A= > FM10K_WRITE_REG(hw, FM10K_INT_MAP(fm10k_int_Mailbox), int_map);=0A= > FM10K_WRITE_REG(hw, FM10K_INT_MAP(fm10k_int_PCIeFault), int_map);=0A= > @@ -2103,7 +2124,7 @@ fm10k_dev_disable_intr_pf(struct rte_eth_dev *dev)= =0A= > struct fm10k_hw *hw =3D FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private)= ;=0A= > uint32_t int_map =3D FM10K_INT_MAP_DISABLE;=0A= > =0A= > - int_map |=3D 0;=0A= > + int_map |=3D FM10K_MISC_VEC_ID;=0A= > =0A= > FM10K_WRITE_REG(hw, FM10K_INT_MAP(fm10k_int_Mailbox), int_map);=0A= > FM10K_WRITE_REG(hw, FM10K_INT_MAP(fm10k_int_PCIeFault), int_map);=0A= > @@ -2134,7 +2155,7 @@ fm10k_dev_enable_intr_vf(struct rte_eth_dev *dev)= =0A= > uint32_t int_map =3D FM10K_INT_MAP_IMMEDIATE;=0A= > =0A= > /* Bind all local non-queue interrupt to vector 0 */=0A= > - int_map |=3D 0;=0A= > + int_map |=3D FM10K_MISC_VEC_ID;=0A= > =0A= > /* Only INT 0 available, other 15 are reserved. */=0A= > FM10K_WRITE_REG(hw, FM10K_VFINT_MAP, int_map);=0A= > @@ -2151,7 +2172,7 @@ fm10k_dev_disable_intr_vf(struct rte_eth_dev *dev)= =0A= > struct fm10k_hw *hw =3D FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private)= ;=0A= > uint32_t int_map =3D FM10K_INT_MAP_DISABLE;=0A= > =0A= > - int_map |=3D 0;=0A= > + int_map |=3D FM10K_MISC_VEC_ID;=0A= > =0A= > /* Only INT 0 available, other 15 are reserved. */=0A= > FM10K_WRITE_REG(hw, FM10K_VFINT_MAP, int_map);=0A= > @@ -2162,6 +2183,64 @@ fm10k_dev_disable_intr_vf(struct rte_eth_dev *dev)= =0A= > }=0A= > =0A= > static int=0A= > +fm10k_dev_rxq_interrupt_setup(struct rte_eth_dev *dev)=0A= > +{=0A= > + struct fm10k_hw *hw =3D FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private)= ;=0A= > + struct rte_intr_handle *intr_handle =3D &dev->pci_dev->intr_handle;=0A= > + uint32_t intr_vector, vec;=0A= > + uint16_t queue_id;=0A= > + int result =3D 0;=0A= > +=0A= > + /* fm10k needs one separate interrupt for mailbox,=0A= > + * so only drivers which support multiple interrupt vectors=0A= > + * e.g. vfio-pci can work for fm10k interrupt mode=0A= > + */=0A= > + if (!rte_intr_cap_multiple(intr_handle) ||=0A= > + dev->data->dev_conf.intr_conf.rxq =3D=3D 0)=0A= > + return result;=0A= > +=0A= > + intr_vector =3D dev->data->nb_rx_queues;=0A= > +=0A= > + /* disable interrupt first */=0A= > + rte_intr_disable(&dev->pci_dev->intr_handle);=0A= > + if (hw->mac.type =3D=3D fm10k_mac_pf)=0A= > + fm10k_dev_disable_intr_pf(dev);=0A= > + else=0A= > + fm10k_dev_disable_intr_vf(dev);=0A= > +=0A= > + if (rte_intr_efd_enable(intr_handle, intr_vector)) {=0A= > + PMD_INIT_LOG(ERR, "Failed to init event fd");=0A= > + result =3D -EIO;=0A= > + }=0A= > +=0A= > + if (rte_intr_dp_is_en(intr_handle) && !result) {=0A= > + intr_handle->intr_vec =3D rte_zmalloc("intr_vec",=0A= > + dev->data->nb_rx_queues * sizeof(int), 0);=0A= > + if (intr_handle->intr_vec) {=0A= > + for (queue_id =3D 0, vec =3D FM10K_RX_VEC_START;=0A= > + queue_id < dev->data->nb_rx_queues;=0A= > + queue_id++) {=0A= > + intr_handle->intr_vec[queue_id] =3D vec;=0A= > + if (vec < intr_handle->nb_efd - 1 + FM10K_RX_VEC_START)=0A= > + vec++;=0A= > + }=0A= > + } else {=0A= > + PMD_INIT_LOG(ERR, "Failed to allocate %d rx_queues"=0A= > + " intr_vec", dev->data->nb_rx_queues);=0A= =0A= Here if failure, should we call rte_intr_efd_disable()?=0A= =0A= Thanks,=0A= Michael=0A= > + result =3D -ENOMEM;=0A= > + }=0A= > + }=0A= > +=0A= > + if (hw->mac.type =3D=3D fm10k_mac_pf)=0A= > + fm10k_dev_enable_intr_pf(dev);=0A= > + else=0A= > + fm10k_dev_enable_intr_vf(dev);=0A= > + rte_intr_enable(&dev->pci_dev->intr_handle);=0A= > + hw->mac.ops.update_int_moderator(hw);=0A= > + return result;=0A= > +}=0A= > +=0A= > +static int=0A= > fm10k_dev_handle_fault(struct fm10k_hw *hw, uint32_t eicr)=0A= > {=0A= > struct fm10k_fault fault;=0A= > @@ -2531,7 +2610,7 @@ static int=0A= > eth_fm10k_dev_init(struct rte_eth_dev *dev)=0A= > {=0A= > struct fm10k_hw *hw =3D FM10K_DEV_PRIVATE_TO_HW(dev->data->dev_private)= ;=0A= > - int diag;=0A= > + int diag, i;=0A= > struct fm10k_macvlan_filter_info *macvlan;=0A= > =0A= > PMD_INIT_FUNC_TRACE();=0A= > @@ -2637,7 +2716,7 @@ eth_fm10k_dev_init(struct rte_eth_dev *dev)=0A= > fm10k_dev_enable_intr_vf(dev);=0A= > }=0A= > =0A= > - /* Enable uio intr after callback registered */=0A= > + /* Enable intr after callback registered */=0A= > rte_intr_enable(&(dev->pci_dev->intr_handle));=0A= > =0A= > hw->mac.ops.update_int_moderator(hw);=0A= > @@ -2645,7 +2724,6 @@ eth_fm10k_dev_init(struct rte_eth_dev *dev)=0A= > /* Make sure Switch Manager is ready before going forward. */=0A= > if (hw->mac.type =3D=3D fm10k_mac_pf) {=0A= > int switch_ready =3D 0;=0A= > - int i;=0A= > =0A= > for (i =3D 0; i < MAX_QUERY_SWITCH_STATE_TIMES; i++) {=0A= > fm10k_mbx_lock(hw);=0A= > @@ -2752,7 +2830,8 @@ static struct eth_driver rte_pmd_fm10k =3D {=0A= > .pci_drv =3D {=0A= > .name =3D "rte_pmd_fm10k",=0A= > .id_table =3D pci_id_fm10k_map,=0A= > - .drv_flags =3D RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_DETACHABLE,=0A= > + .drv_flags =3D RTE_PCI_DRV_NEED_MAPPING | RTE_PCI_DRV_INTR_LSC |=0A= > + RTE_PCI_DRV_DETACHABLE,=0A= > },=0A= > .eth_dev_init =3D eth_fm10k_dev_init,=0A= > .eth_dev_uninit =3D eth_fm10k_dev_uninit,=0A= =0A=