From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2110442FAD; Tue, 1 Aug 2023 04:21:05 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id AAC3940A7D; Tue, 1 Aug 2023 04:21:04 +0200 (CEST) Received: from mgamail.intel.com (unknown [192.55.52.43]) by mails.dpdk.org (Postfix) with ESMTP id EC1FD400D5 for ; Tue, 1 Aug 2023 04:21:02 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1690856463; x=1722392463; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=hP9iZl5ZBmtzFYN+Zc8CZtyCpk2Z/mVMb1jasloxO4k=; b=Aitr31DE0lej24ph839hX20ezqj7LmG38UoJqHFge8tsrK7CMF/PR4ah U9CkjQlHURae3EqDwrnFZl9FfsUWKokObesc0bQIH8UrSAmC54iKqO64v hKFQD5OLBIvEiGCcD1B0DUKHqVpHY2PseozzS6GUWXDbw1nub1u2mX0Dh 7nl5OMOuB7pE+GpIl89w0D9fxsCFUJFX/hjszIxi20H/q2Bb3yBn7BWCY zTIZbjXXU4F0qmRHAGdsOycNaQ0arjoIsspy5mYdvG0qFR/y3y/TbxvGA zfr04GtIyBYzWCJBHXU86ZgGHDbkzE/9L7+W5M6/DeNjtkj39nYaZ3LqU g==; X-IronPort-AV: E=McAfee;i="6600,9927,10788"; a="455552318" X-IronPort-AV: E=Sophos;i="6.01,246,1684825200"; d="scan'208";a="455552318" Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jul 2023 19:20:59 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10788"; a="731786562" X-IronPort-AV: E=Sophos;i="6.01,246,1684825200"; d="scan'208";a="731786562" Received: from unknown (HELO root..) ([10.239.252.115]) by fmsmga007-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 31 Jul 2023 19:20:56 -0700 From: Shiyang He To: dev@dpdk.org Cc: yidingx.zhou@intel.com, Shiyang He , Simei Su , Wenjun Wu , Qiming Yang , Yuying Zhang , Beilei Xing , Jingjing Wu Subject: [PATCH] net/iavf: add devargs to enable vf auto-reset Date: Tue, 1 Aug 2023 09:48:14 +0000 Message-Id: <20230801094815.4028257-1-shiyangx.he@intel.com> X-Mailer: git-send-email 2.37.2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Originally, the iavf PMD does not perform special actions when it receives a PF-to-VF reset event, resulting in vf being offline and unavailable. This patch enables vf auto-reset by setting 'watchdog_period' devargs to true. The iavf PMD will perform an automatic reset to bring the vf back online when it receives a PF-to-VF event. Signed-off-by: Shiyang He --- doc/guides/nics/intel_vf.rst | 3 + drivers/net/iavf/iavf.h | 28 ++++++ drivers/net/iavf/iavf_ethdev.c | 166 +++++++++++++++++++++++++++++++++ drivers/net/iavf/iavf_rxtx.c | 25 +++++ drivers/net/iavf/iavf_rxtx.h | 1 + drivers/net/iavf/iavf_vchnl.c | 8 ++ 6 files changed, 231 insertions(+) diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst index d365dbc185..c0acd2a7f5 100644 --- a/doc/guides/nics/intel_vf.rst +++ b/doc/guides/nics/intel_vf.rst @@ -101,6 +101,9 @@ For more detail on SR-IOV, please refer to the following documents: Set ``devargs`` parameter ``watchdog_period`` to adjust the watchdog period in microseconds, or set it to 0 to disable the watchdog, for example, ``-a 18:01.0,watchdog_period=5000`` or ``-a 18:01.0,watchdog_period=0``. + Enable vf auto-reset by setting the ``devargs`` parameter like ``-a 18:01.0,enable_auto_reset=1`` when IAVF is backed + by an Intel® E810 device or an Intel® 700 Series Ethernet device. + The PCIE host-interface of Intel Ethernet Switch FM10000 Series VF infrastructure ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/drivers/net/iavf/iavf.h b/drivers/net/iavf/iavf.h index 98861e4242..a150b7d5cf 100644 --- a/drivers/net/iavf/iavf.h +++ b/drivers/net/iavf/iavf.h @@ -95,6 +95,24 @@ #define IAVF_L2TPV2_FLAGS_LEN 0x4000 +#ifndef DIV_ROUND_UP +#define DIV_ROUND_UP(n, d) ( \ +{ \ + const typeof(d) __d = d; \ + (((n) + (__d) - 1) / (__d));\ +} \ +) +#endif +#ifndef DELAY +#define DELAY(x) rte_delay_us(x) +#endif +#ifndef msleep +#define msleep(x) DELAY(1000 * (x)) +#endif +#ifndef usleep_range +#define usleep_range(min, max) msleep(DIV_ROUND_UP(min, 1000)) +#endif + struct iavf_adapter; struct iavf_rx_queue; struct iavf_tx_queue; @@ -305,6 +323,7 @@ struct iavf_devargs { uint8_t proto_xtr[IAVF_MAX_QUEUE_NUM]; uint16_t quanta_size; uint32_t watchdog_period; + uint8_t enable_auto_reset; }; struct iavf_security_ctx; @@ -424,6 +443,14 @@ _atomic_set_async_response_cmd(struct iavf_info *vf, enum virtchnl_ops ops) return !ret; } + +static inline bool +iavf_is_reset(struct iavf_hw *hw) +{ + return !(IAVF_READ_REG(hw, IAVF_VF_ARQLEN1) & + IAVF_VF_ARQLEN1_ARQENABLE_MASK); +} + int iavf_check_api_version(struct iavf_adapter *adapter); int iavf_get_vf_resource(struct iavf_adapter *adapter); void iavf_dev_event_handler_fini(void); @@ -501,4 +528,5 @@ int iavf_flow_sub_check(struct iavf_adapter *adapter, struct iavf_fsub_conf *filter); void iavf_dev_watchdog_enable(struct iavf_adapter *adapter); void iavf_dev_watchdog_disable(struct iavf_adapter *adapter); +int iavf_handle_hw_reset(struct rte_eth_dev *dev); #endif /* _IAVF_ETHDEV_H_ */ diff --git a/drivers/net/iavf/iavf_ethdev.c b/drivers/net/iavf/iavf_ethdev.c index f2fc5a5621..5f9fce34a5 100644 --- a/drivers/net/iavf/iavf_ethdev.c +++ b/drivers/net/iavf/iavf_ethdev.c @@ -37,6 +37,7 @@ #define IAVF_PROTO_XTR_ARG "proto_xtr" #define IAVF_QUANTA_SIZE_ARG "quanta_size" #define IAVF_RESET_WATCHDOG_ARG "watchdog_period" +#define IAVF_ENABLE_AUTO_RESET_ARG "enable_auto_reset" uint64_t iavf_timestamp_dynflag; int iavf_timestamp_dynfield_offset = -1; @@ -45,6 +46,7 @@ static const char * const iavf_valid_args[] = { IAVF_PROTO_XTR_ARG, IAVF_QUANTA_SIZE_ARG, IAVF_RESET_WATCHDOG_ARG, + IAVF_ENABLE_AUTO_RESET_ARG, NULL }; @@ -305,6 +307,8 @@ iavf_dev_watchdog(void *cb_arg) adapter->vf.vf_reset = true; adapter->vf.link_up = false; + iavf_handle_hw_reset(adapter->vf.eth_dev); + rte_eth_dev_callback_process(adapter->vf.eth_dev, RTE_ETH_EVENT_INTR_RESET, NULL); } @@ -2210,6 +2214,26 @@ parse_u16(__rte_unused const char *key, const char *value, void *args) return 0; } +static int +parse_bool(const char *key, const char *value, void *args) +{ + int *i = (int *)args; + char *end; + int num; + + num = strtoul(value, &end, 10); + + if (num != 0 && num != 1) { + PMD_DRV_LOG(WARNING, "invalid value:\"%s\" for key:\"%s\", " + "value must be 0 or 1", + value, key); + return -1; + } + + *i = num; + return 0; +} + static int iavf_parse_watchdog_period(__rte_unused const char *key, const char *value, void *args) { @@ -2278,6 +2302,11 @@ static int iavf_parse_devargs(struct rte_eth_dev *dev) goto bail; } + ret = rte_kvargs_process(kvlist, IAVF_ENABLE_AUTO_RESET_ARG, + &parse_bool, &ad->devargs.enable_auto_reset); + if (ret) + goto bail; + bail: rte_kvargs_free(kvlist); return ret; @@ -2879,6 +2908,143 @@ iavf_dev_reset(struct rte_eth_dev *dev) return iavf_dev_init(dev); } +static bool +iavf_is_reset_detected(struct iavf_adapter *adapter) +{ + struct iavf_hw *hw = &adapter->hw; + int i; + + /* poll until we see the reset actually happen */ + for (i = 0; i < IAVF_RESET_WAIT_CNT; i++) { + if (iavf_is_reset(hw)) + return true; + usleep_range(5000, 10000); + } + + return false; +} + +static int +iavf_uninit_hw(struct rte_eth_dev *dev, struct iavf_hw *hw) +{ + struct rte_pci_device *pci_dev = RTE_ETH_DEV_TO_PCI(dev); + struct rte_intr_handle *intr_handle = dev->intr_handle; + struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); + struct iavf_adapter *adapter = dev->data->dev_private; + + iavf_reset_queues(dev); + + /* Disable the interrupt for Rx */ + rte_intr_efd_disable(intr_handle); + /* Rx interrupt vector mapping free */ + rte_intr_vec_list_free(intr_handle); + + adapter->stopped = 1; + dev->data->dev_started = 0; + + adapter->closed = true; + + /* free iAVF security device context all related resources */ + iavf_security_ctx_destroy(adapter); + + iavf_flow_flush(dev, NULL); + iavf_flow_uninit(adapter); + + if (vf->promisc_unicast_enabled || vf->promisc_multicast_enabled) + iavf_config_promisc(adapter, false, false); + + iavf_shutdown_adminq(hw); + if (vf->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_WB_ON_ITR) { + /* disable uio intr before callback unregister */ + rte_intr_disable(intr_handle); + + /* unregister callback func from eal lib */ + rte_intr_callback_unregister(intr_handle, + iavf_dev_interrupt_handler, dev); + } + iavf_disable_irq0(hw); + + if (vf->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_QOS) + iavf_tm_conf_uninit(dev); + + if (vf->vf_res->vf_cap_flags & VIRTCHNL_VF_OFFLOAD_RSS_PF) { + if (vf->rss_lut) { + rte_free(vf->rss_lut); + vf->rss_lut = NULL; + } + if (vf->rss_key) { + rte_free(vf->rss_key); + vf->rss_key = NULL; + } + } + + rte_free(vf->vf_res); + vf->vsi_res = NULL; + vf->vf_res = NULL; + + rte_free(vf->aq_resp); + vf->aq_resp = NULL; + + if (vf->vf_reset && !rte_pci_set_bus_master(pci_dev, true)) + vf->vf_reset = false; + + /* disable watchdog */ + iavf_dev_watchdog_disable(adapter); + + return 0; +} + +static int +iavf_reset_hw(struct rte_eth_dev *dev, struct iavf_hw *hw) +{ + iavf_uninit_hw(dev, hw); + + return iavf_dev_init(dev); +} + +/* + * Handle hardware reset + */ +int +iavf_handle_hw_reset(struct rte_eth_dev *dev) +{ + struct iavf_adapter *adapter = dev->data->dev_private; + struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); + int ret; + + if (!adapter->devargs.enable_auto_reset) + return 0; + + if (!iavf_is_reset_detected(adapter)) + return 0; + + if (iavf_reset_hw(dev, hw)) + return -1; + + /* + * This may return a failure due to the vf may not able to get + * a response from the pf. + */ + ret = iavf_dev_configure(dev); + if (ret) + PMD_DRV_LOG(ERR, "Failed to configure dev"); + + iavf_dev_xstats_reset(dev); + + /* start the device */ + ret = iavf_dev_start(dev); + if (ret) { + PMD_DRV_LOG(ERR, "Failed to start dev"); + return ret; + } + + dev->data->dev_started = 1; + vf->vf_reset = false; + + return 0; +} + static int iavf_dcf_cap_check_handler(__rte_unused const char *key, const char *value, __rte_unused void *opaque) diff --git a/drivers/net/iavf/iavf_rxtx.c b/drivers/net/iavf/iavf_rxtx.c index f7df4665d1..a1f556d1ce 100644 --- a/drivers/net/iavf/iavf_rxtx.c +++ b/drivers/net/iavf/iavf_rxtx.c @@ -1094,6 +1094,31 @@ iavf_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid) rte_free(q); } +void +iavf_reset_queues(struct rte_eth_dev *dev) +{ + struct iavf_rx_queue *rxq; + struct iavf_tx_queue *txq; + int i; + + for (i = 0; i < dev->data->nb_tx_queues; i++) { + txq = dev->data->tx_queues[i]; + if (!txq) + continue; + iavf_txq_release_mbufs_ops[txq->rel_mbufs_type].release_mbufs(txq); + reset_tx_queue(txq); + dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED; + } + for (i = 0; i < dev->data->nb_rx_queues; i++) { + rxq = dev->data->rx_queues[i]; + if (!rxq) + continue; + iavf_rxq_release_mbufs_ops[rxq->rel_mbufs_type].release_mbufs(rxq); + reset_rx_queue(rxq); + dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED; + } +} + void iavf_stop_queues(struct rte_eth_dev *dev) { diff --git a/drivers/net/iavf/iavf_rxtx.h b/drivers/net/iavf/iavf_rxtx.h index 605ea3f824..237022338e 100644 --- a/drivers/net/iavf/iavf_rxtx.h +++ b/drivers/net/iavf/iavf_rxtx.h @@ -650,6 +650,7 @@ int iavf_dev_tx_queue_start(struct rte_eth_dev *dev, uint16_t tx_queue_id); int iavf_dev_tx_queue_stop(struct rte_eth_dev *dev, uint16_t tx_queue_id); int iavf_dev_tx_done_cleanup(void *txq, uint32_t free_cnt); void iavf_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid); +void iavf_reset_queues(struct rte_eth_dev *dev); void iavf_stop_queues(struct rte_eth_dev *dev); uint16_t iavf_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts); diff --git a/drivers/net/iavf/iavf_vchnl.c b/drivers/net/iavf/iavf_vchnl.c index 524732f67d..19e22ae283 100644 --- a/drivers/net/iavf/iavf_vchnl.c +++ b/drivers/net/iavf/iavf_vchnl.c @@ -80,6 +80,14 @@ iavf_dev_event_handle(void *param __rte_unused) TAILQ_FOREACH_SAFE(pos, &pending, next, save_next) { TAILQ_REMOVE(&pending, pos, next); rte_eth_dev_callback_process(pos->dev, pos->event, pos->param); + + switch (pos->event) { + case RTE_ETH_EVENT_INTR_RESET: + iavf_handle_hw_reset(pos->dev); + break; + default: + break; + } rte_free(pos); } } -- 2.37.2