From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id B174A4263D; Tue, 26 Sep 2023 07:05:46 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 9A641402A9; Tue, 26 Sep 2023 07:05:46 +0200 (CEST) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.31]) by mails.dpdk.org (Postfix) with ESMTP id C843240271 for ; Tue, 26 Sep 2023 07:05:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695704744; x=1727240744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UCpjJn77oyDLDuFgR+HhObz99hI27N6VdwBaVXm1GJc=; b=T0HLzuZZiCUeSIKFlPviWcGOijp4HqjPMX0A1bKp2brB2lWadLGUiUri MyXzmuw0pQiMqUA9VZhCmr5avmjQ+N9NzWju+8NPYjbbFlpEUBjSezp9Z Uhk7Dmfle+YDphYdFx6zN/RqktTWwq0xjQeOCINKwKvpjwQ8vKBJqChBT aFqc5K6piEVUagW5Yu4QLgT9cCMz7jYisW/uaEvQ8fSyhlu+5xZ5KQ5J1 HP1DKLz7dzFHpi2lsKsPNRoFmV9Tf/QSzHSj+Y9n8s9jC3jIF7NyA4Dx2 E/KVPaAE+fQzY4nwL2Jnsv8iAp3FNHIvGuZWgdMmoELJ0YyMWIR+Jedh7 A==; X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="445616552" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="445616552" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Sep 2023 22:05:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="922285057" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="922285057" Received: from unknown (HELO root..) ([10.239.252.115]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Sep 2023 22:05:39 -0700 From: Shiyang He To: dev@dpdk.org Cc: yidingx.zhou@intel.com, Shiyang He , Qi Zhang , Liang-Min Larry Wang , Qiming Yang , Wenjun Wu , Simei Su , Yuying Zhang , Beilei Xing , Jingjing Wu Subject: [PATCH v5] net/iavf: add devargs to enable vf auto-reset Date: Tue, 26 Sep 2023 12:31:29 +0000 Message-Id: <20230926123129.3878749-1-shiyangx.he@intel.com> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20230926113137.3861198-1-shiyangx.he@intel.com> References: <20230926113137.3861198-1-shiyangx.he@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Originally, the iavf PMD does not perform special actions when it receives a PF-to-VF reset event, resulting in vf being offline and unavailable. This patch enables vf auto-reset by setting 'watchdog_period' devargs to true. The iavf PMD will perform an automatic reset to bring the vf back online when it receives a PF-to-VF event. v2: handling reset by event handler v3: change reset process v4: rebase v5: change enable_auto_reset to enable_reset Acked-by: Qi Zhang Signed-off-by: Shiyang He Signed-off-by: Liang-Min Larry Wang --- doc/guides/nics/intel_vf.rst | 3 + doc/guides/rel_notes/release_23_11.rst | 3 + drivers/net/iavf/iavf.h | 7 +++ drivers/net/iavf/iavf_ethdev.c | 86 +++++++++++++++++++++++--- drivers/net/iavf/iavf_rxtx.c | 52 ++++++++++------ drivers/net/iavf/iavf_vchnl.c | 11 +++- 6 files changed, 135 insertions(+), 27 deletions(-) diff --git a/doc/guides/nics/intel_vf.rst b/doc/guides/nics/intel_vf.rst index d365dbc185..f771948242 100644 --- a/doc/guides/nics/intel_vf.rst +++ b/doc/guides/nics/intel_vf.rst @@ -101,6 +101,9 @@ For more detail on SR-IOV, please refer to the following documents: Set ``devargs`` parameter ``watchdog_period`` to adjust the watchdog period in microseconds, or set it to 0 to disable the watchdog, for example, ``-a 18:01.0,watchdog_period=5000`` or ``-a 18:01.0,watchdog_period=0``. + Enable vf auto-reset by setting the ``devargs`` parameter like ``-a 18:01.0,enable_reset=1`` when IAVF is backed + by an Intel® E810 device or an Intel® 700 Series Ethernet device. + The PCIE host-interface of Intel Ethernet Switch FM10000 Series VF infrastructure ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/doc/guides/rel_notes/release_23_11.rst b/doc/guides/rel_notes/release_23_11.rst index 03191b3ae0..8536ce88f4 100644 --- a/doc/guides/rel_notes/release_23_11.rst +++ b/doc/guides/rel_notes/release_23_11.rst @@ -86,6 +86,9 @@ New Features * Added support for port representor. +* **Updated Intel iavf driver.** + * Added support for iavf auto-reset. + Removed Items ------------- diff --git a/drivers/net/iavf/iavf.h b/drivers/net/iavf/iavf.h index 98861e4242..be6b9920e6 100644 --- a/drivers/net/iavf/iavf.h +++ b/drivers/net/iavf/iavf.h @@ -277,6 +277,8 @@ struct iavf_info { struct rte_eth_dev *eth_dev; + bool in_reset_recovery; + uint32_t ptp_caps; rte_spinlock_t phc_time_aq_lock; }; @@ -305,6 +307,7 @@ struct iavf_devargs { uint8_t proto_xtr[IAVF_MAX_QUEUE_NUM]; uint16_t quanta_size; uint32_t watchdog_period; + uint8_t enable_reset; }; struct iavf_security_ctx; @@ -426,6 +429,9 @@ _atomic_set_async_response_cmd(struct iavf_info *vf, enum virtchnl_ops ops) } int iavf_check_api_version(struct iavf_adapter *adapter); int iavf_get_vf_resource(struct iavf_adapter *adapter); +void iavf_dev_event_post(struct rte_eth_dev *dev, + enum rte_eth_event_type event, + void *param, size_t param_alloc_size); void iavf_dev_event_handler_fini(void); int iavf_dev_event_handler_init(void); void iavf_handle_virtchnl_msg(struct rte_eth_dev *dev); @@ -501,4 +507,5 @@ int iavf_flow_sub_check(struct iavf_adapter *adapter, struct iavf_fsub_conf *filter); void iavf_dev_watchdog_enable(struct iavf_adapter *adapter); void iavf_dev_watchdog_disable(struct iavf_adapter *adapter); +int iavf_handle_hw_reset(struct rte_eth_dev *dev); #endif /* _IAVF_ETHDEV_H_ */ diff --git a/drivers/net/iavf/iavf_ethdev.c b/drivers/net/iavf/iavf_ethdev.c index 97390237ba..191870c06b 100644 --- a/drivers/net/iavf/iavf_ethdev.c +++ b/drivers/net/iavf/iavf_ethdev.c @@ -37,6 +37,7 @@ #define IAVF_PROTO_XTR_ARG "proto_xtr" #define IAVF_QUANTA_SIZE_ARG "quanta_size" #define IAVF_RESET_WATCHDOG_ARG "watchdog_period" +#define IAVF_ENABLE_AUTO_RESET_ARG "enable_reset" uint64_t iavf_timestamp_dynflag; int iavf_timestamp_dynfield_offset = -1; @@ -45,6 +46,7 @@ static const char * const iavf_valid_args[] = { IAVF_PROTO_XTR_ARG, IAVF_QUANTA_SIZE_ARG, IAVF_RESET_WATCHDOG_ARG, + IAVF_ENABLE_AUTO_RESET_ARG, NULL }; @@ -307,8 +309,8 @@ iavf_dev_watchdog(void *cb_arg) adapter->vf.vf_reset = true; adapter->vf.link_up = false; - rte_eth_dev_callback_process(adapter->vf.eth_dev, - RTE_ETH_EVENT_INTR_RESET, NULL); + iavf_dev_event_post(adapter->vf.eth_dev, RTE_ETH_EVENT_INTR_RESET, + NULL, 0); } } @@ -1101,12 +1103,15 @@ iavf_dev_stop(struct rte_eth_dev *dev) /* Rx interrupt vector mapping free */ rte_intr_vec_list_free(intr_handle); - /* remove all mac addrs */ - iavf_add_del_all_mac_addr(adapter, false); + /* adminq will be disabled when vf is resetting. */ + if (!vf->in_reset_recovery) { + /* remove all mac addrs */ + iavf_add_del_all_mac_addr(adapter, false); - /* remove all multicast addresses */ - iavf_add_del_mc_addr_list(adapter, vf->mc_addrs, vf->mc_addrs_num, + /* remove all multicast addresses */ + iavf_add_del_mc_addr_list(adapter, vf->mc_addrs, vf->mc_addrs_num, false); + } iavf_stop_queues(dev); @@ -2239,6 +2244,26 @@ parse_u16(__rte_unused const char *key, const char *value, void *args) return 0; } +static int +parse_bool(const char *key, const char *value, void *args) +{ + int *i = (int *)args; + char *end; + int num; + + num = strtoul(value, &end, 10); + + if (num != 0 && num != 1) { + PMD_DRV_LOG(WARNING, "invalid value:\"%s\" for key:\"%s\", " + "value must be 0 or 1", + value, key); + return -1; + } + + *i = num; + return 0; +} + static int iavf_parse_watchdog_period(__rte_unused const char *key, const char *value, void *args) { @@ -2307,6 +2332,11 @@ static int iavf_parse_devargs(struct rte_eth_dev *dev) goto bail; } + ret = rte_kvargs_process(kvlist, IAVF_ENABLE_AUTO_RESET_ARG, + &parse_bool, &ad->devargs.enable_reset); + if (ret) + goto bail; + bail: rte_kvargs_free(kvlist); return ret; @@ -2887,12 +2917,15 @@ iavf_dev_close(struct rte_eth_dev *dev) static int iavf_dev_uninit(struct rte_eth_dev *dev) { + struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); + if (rte_eal_process_type() != RTE_PROC_PRIMARY) return -EPERM; iavf_dev_close(dev); - iavf_dev_event_handler_fini(); + if (!vf->in_reset_recovery) + iavf_dev_event_handler_fini(); return 0; } @@ -2905,6 +2938,7 @@ iavf_dev_reset(struct rte_eth_dev *dev) { int ret; struct iavf_hw *hw = IAVF_DEV_PRIVATE_TO_HW(dev->data->dev_private); + struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); /* * Check whether the VF reset has been done and inform application, @@ -2916,6 +2950,7 @@ iavf_dev_reset(struct rte_eth_dev *dev) PMD_DRV_LOG(ERR, "Wait too long for reset done!\n"); return ret; } + vf->vf_reset = false; PMD_DRV_LOG(DEBUG, "Start dev_reset ...\n"); ret = iavf_dev_uninit(dev); @@ -2925,6 +2960,43 @@ iavf_dev_reset(struct rte_eth_dev *dev) return iavf_dev_init(dev); } +/* + * Handle hardware reset + */ +int +iavf_handle_hw_reset(struct rte_eth_dev *dev) +{ + struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); + int ret; + + vf->in_reset_recovery = true; + + ret = iavf_dev_reset(dev); + if (ret) + goto error; + + /* VF states restore */ + ret = iavf_dev_configure(dev); + if (ret) + goto error; + + iavf_dev_xstats_reset(dev); + + /* start the device */ + ret = iavf_dev_start(dev); + if (ret) + goto error; + dev->data->dev_started = 1; + + vf->in_reset_recovery = false; + return 0; + +error: + PMD_DRV_LOG(DEBUG, "RESET recover with error code=%d\n", ret); + vf->in_reset_recovery = false; + return ret; +} + static int iavf_dcf_cap_check_handler(__rte_unused const char *key, const char *value, __rte_unused void *opaque) diff --git a/drivers/net/iavf/iavf_rxtx.c b/drivers/net/iavf/iavf_rxtx.c index 72e8ae8aa6..0484988d13 100644 --- a/drivers/net/iavf/iavf_rxtx.c +++ b/drivers/net/iavf/iavf_rxtx.c @@ -1101,15 +1101,44 @@ iavf_dev_tx_queue_release(struct rte_eth_dev *dev, uint16_t qid) rte_free(q); } +static void +iavf_reset_queues(struct rte_eth_dev *dev) +{ + struct iavf_rx_queue *rxq; + struct iavf_tx_queue *txq; + int i; + + for (i = 0; i < dev->data->nb_tx_queues; i++) { + txq = dev->data->tx_queues[i]; + if (!txq) + continue; + iavf_txq_release_mbufs_ops[txq->rel_mbufs_type].release_mbufs(txq); + reset_tx_queue(txq); + dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED; + } + for (i = 0; i < dev->data->nb_rx_queues; i++) { + rxq = dev->data->rx_queues[i]; + if (!rxq) + continue; + iavf_rxq_release_mbufs_ops[rxq->rel_mbufs_type].release_mbufs(rxq); + reset_rx_queue(rxq); + dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED; + } +} + void iavf_stop_queues(struct rte_eth_dev *dev) { struct iavf_adapter *adapter = IAVF_DEV_PRIVATE_TO_ADAPTER(dev->data->dev_private); struct iavf_info *vf = IAVF_DEV_PRIVATE_TO_VF(dev->data->dev_private); - struct iavf_rx_queue *rxq; - struct iavf_tx_queue *txq; - int ret, i; + int ret; + + /* adminq will be disabled when vf is resetting. */ + if (vf->in_reset_recovery) { + iavf_reset_queues(dev); + return; + } /* Stop All queues */ if (!vf->lv_enabled) { @@ -1125,22 +1154,7 @@ iavf_stop_queues(struct rte_eth_dev *dev) if (ret) PMD_DRV_LOG(WARNING, "Fail to stop queues"); - for (i = 0; i < dev->data->nb_tx_queues; i++) { - txq = dev->data->tx_queues[i]; - if (!txq) - continue; - iavf_txq_release_mbufs_ops[txq->rel_mbufs_type].release_mbufs(txq); - reset_tx_queue(txq); - dev->data->tx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED; - } - for (i = 0; i < dev->data->nb_rx_queues; i++) { - rxq = dev->data->rx_queues[i]; - if (!rxq) - continue; - iavf_rxq_release_mbufs_ops[rxq->rel_mbufs_type].release_mbufs(rxq); - reset_rx_queue(rxq); - dev->data->rx_queue_state[i] = RTE_ETH_QUEUE_STATE_STOPPED; - } + iavf_reset_queues(dev); } #define IAVF_RX_FLEX_ERR0_BITS \ diff --git a/drivers/net/iavf/iavf_vchnl.c b/drivers/net/iavf/iavf_vchnl.c index b3e106b41f..9cefa26854 100644 --- a/drivers/net/iavf/iavf_vchnl.c +++ b/drivers/net/iavf/iavf_vchnl.c @@ -79,6 +79,15 @@ iavf_dev_event_handle(void *param __rte_unused) struct iavf_event_element *pos, *save_next; TAILQ_FOREACH_SAFE(pos, &pending, next, save_next) { TAILQ_REMOVE(&pending, pos, next); + + struct iavf_adapter *adapter = pos->dev->data->dev_private; + if (pos->event == RTE_ETH_EVENT_INTR_RESET && + adapter->devargs.enable_reset) { + iavf_handle_hw_reset(pos->dev); + rte_free(pos); + continue; + } + rte_eth_dev_callback_process(pos->dev, pos->event, pos->param); rte_free(pos); } @@ -87,7 +96,7 @@ iavf_dev_event_handle(void *param __rte_unused) return 0; } -static void +void iavf_dev_event_post(struct rte_eth_dev *dev, enum rte_eth_event_type event, void *param, size_t param_alloc_size) -- 2.37.2