From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id B6B92A00E6 for ; Wed, 20 Mar 2019 05:52:16 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id A4F4A1B19; Wed, 20 Mar 2019 05:52:15 +0100 (CET) Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by dpdk.org (Postfix) with ESMTP id 3C0365A for ; Wed, 20 Mar 2019 05:52:13 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga101.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Mar 2019 21:52:11 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,247,1549958400"; d="scan'208";a="156564432" Received: from dpdk51.sh.intel.com ([10.67.110.245]) by fmsmga001.fm.intel.com with ESMTP; 19 Mar 2019 21:52:09 -0700 From: Qi Zhang To: thomas@monjalon.net, declan.doherty@intel.com, ferruh.yigit@intel.com Cc: ktraynor@redhat.com, sthemmin@microsoft.com, dev@dpdk.org, benjamin.h.shelton@intel.com, narender.vangati@intel.com, Qi Zhang Date: Wed, 20 Mar 2019 12:54:03 +0800 Message-Id: <20190320045403.14594-1-qi.z.zhang@intel.com> X-Mailer: git-send-email 2.13.6 Subject: [dpdk-dev] [PATCH] ethdev: claim device reset as async X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Content-Type: text/plain; charset="UTF-8" Message-ID: <20190320045403.GKGix5eSRf1MEzMukkAr9innsDjYqZVkOGLfk_8TJa0@z> Device reset should be implemented in an async way since it is possible to be invoked in interrupt thread and sometimes to reset a device need to wait for some dependency, for example, a VF expects for PF ready or a NIC function as part of a SOC wait for the whole system reset complete, and all these time-consuming tasks will block the interrupt thread. The patch rename rte_eth_dev_reset to rte_eth_dev_reset_async and rework the implementation. It will spawn a new thread which will call ops->dev_reset, and when finished it will raise the event RTE_ETH_EVENT_RESET_COMPLETE. The application should always wait for this event before it continues to configure and restart the device. Signed-off-by: Qi Zhang --- RFC3 -> v1 - fix state check in rte_eth_find_next and rte_eth_find_next_owned_by - fix compile error in netvsc - update document and release notes - improve comments and couple typo fix app/test-pmd/testpmd.c | 50 +++++++++++++++++++++++++++- doc/guides/prog_guide/poll_mode_drv.rst | 34 ++++++++++++------- doc/guides/rel_notes/release_19_05.rst | 6 ++++ drivers/net/ixgbe/ixgbe_ethdev.c | 4 +-- drivers/net/netvsc/hn_var.h | 2 +- drivers/net/netvsc/hn_vf.c | 4 +-- lib/librte_ethdev/rte_ethdev.c | 56 +++++++++++++++++++++++++++++--- lib/librte_ethdev/rte_ethdev.h | 50 +++++++++++++++++----------- lib/librte_ethdev/rte_ethdev_version.map | 7 ++++ 9 files changed, 171 insertions(+), 42 deletions(-) diff --git a/app/test-pmd/testpmd.c b/app/test-pmd/testpmd.c index d9d0c16d4..27ddfdb59 100644 --- a/app/test-pmd/testpmd.c +++ b/app/test-pmd/testpmd.c @@ -2232,6 +2232,54 @@ close_port(portid_t pid) printf("Done\n"); } +static int reset_ret; +static int reset_done; + +static int +on_reset_complete(__rte_unused uint16_t port_id, + __rte_unused enum rte_eth_event_type event, + __rte_unused void *cb_arg, + void *ret_param) +{ + RTE_ASSERT(event == RTE_ETH_EVENT_RESET_COMPLETE); + + reset_ret = *(int *)ret_param; + reset_done = 1; + return 0; +} + +static int +do_dev_reset_sync(portid_t pid) +{ + int ret; + + ret = rte_eth_dev_callback_register(pid, + RTE_ETH_EVENT_RESET_COMPLETE, + on_reset_complete, NULL); + + if (ret) { + printf("Fail to reigster callback function\n"); + return ret; + } + + reset_done = 0; + ret = rte_eth_dev_reset_async(pid); + if (ret) + goto finish; + + while (!reset_done) + rte_delay_ms(10); + + ret = reset_ret; + +finish: + rte_eth_dev_callback_unregister(pid, + RTE_ETH_EVENT_RESET_COMPLETE, + on_reset_complete, NULL); + + return ret; +} + void reset_port(portid_t pid) { @@ -2260,7 +2308,7 @@ reset_port(portid_t pid) continue; } - diag = rte_eth_dev_reset(pi); + diag = do_dev_reset_sync(pi); if (diag == 0) { port = &ports[pi]; port->need_reconfig = 1; diff --git a/doc/guides/prog_guide/poll_mode_drv.rst b/doc/guides/prog_guide/poll_mode_drv.rst index 6fae39f90..a0aa8fb74 100644 --- a/doc/guides/prog_guide/poll_mode_drv.rst +++ b/doc/guides/prog_guide/poll_mode_drv.rst @@ -581,7 +581,7 @@ NIC Reset API .. code-block:: c - int rte_eth_dev_reset(uint16_t port_id); + int rte_eth_dev_reset_sync(uint16_t port_id); Sometimes a port has to be reset passively. For example when a PF is reset, all its VFs should also be reset by the application to make them @@ -594,7 +594,7 @@ the application should register a callback function to handle these events. When a PMD needs to trigger a reset, it can trigger an RTE_ETH_EVENT_INTR_RESET event. On receiving an RTE_ETH_EVENT_INTR_RESET event, applications can handle it as follows: Stop working queues, stop -calling Rx and Tx functions, and then call rte_eth_dev_reset(). For +calling Rx and Tx functions, and then call rte_eth_dev_reset_sync(). For thread safety all these operations should be called from the same thread. For example when PF is reset, the PF sends a message to notify VFs of @@ -605,13 +605,23 @@ This means that a PF reset triggers an RTE_ETH_EVENT_INTR_RESET event within VFs. The function _rte_eth_dev_callback_process() will call the registered callback function. The callback function can trigger the application to handle all operations the VF reset requires including -stopping Rx/Tx queues and calling rte_eth_dev_reset(). - -The rte_eth_dev_reset() itself is a generic function which only does -some hardware reset operations through calling dev_unint() and -dev_init(), and itself does not handle synchronization, which is handled -by application. - -The PMD itself should not call rte_eth_dev_reset(). The PMD can trigger -the application to handle reset event. It is duty of application to -handle all synchronization before it calls rte_eth_dev_reset(). +stopping Rx/Tx queues and calling rte_eth_dev_reset_async(). + +As the name, rte_eth_dev_reset_async is an async API, it will spwan a +new thread to call ops->dev_reset, once it is finished, it will raise +the RTE_ETH_EVENT_RESET_COMPLETE event to notify application. That makes +things easy for an application that want to reset the device from the +interrupt thread since typically a RTE_ETH_EVENT_INTR_RESET handler is +invoked in interrupt thread. The typical implementation of ops->dev_reset +will do some hardware reset operations through calling dev_uninit() and +dev_init(). + +Application should not assume device reset is finished after +rte_eth_dev_reset_async return, it should always wait for a +RTE_ETH_EVENT_RESET_COMPLETE event and check the reset result. +If reset success, application should call rte_eth_dev_configure(), +rte_eth_rx_queue_setup(), rte_eth_tx_queue_setup(), +and rte_eth_dev_start() to reconfigure the device as appropriate. + +To avoid unexpected behavior, the application should stop calling +Tx and Rx functions before calling rte_eth_dev_reset_async(). diff --git a/doc/guides/rel_notes/release_19_05.rst b/doc/guides/rel_notes/release_19_05.rst index 61a2c7383..4b7fbad39 100644 --- a/doc/guides/rel_notes/release_19_05.rst +++ b/doc/guides/rel_notes/release_19_05.rst @@ -91,6 +91,12 @@ New Features * Added promiscuous mode support. +* **Claim device reset as async API.** + + rte_eth_dev_reset is replaced by rte_eth_dev_reset_async which is implemented + as an async API. Application should wait for RTE_ETH_EVENT_RESET_COMPLETE + event after invoke the API before continues to configure and restart the + device. Removed Items ------------- diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c b/drivers/net/ixgbe/ixgbe_ethdev.c index 97e10217d..cd1543da0 100644 --- a/drivers/net/ixgbe/ixgbe_ethdev.c +++ b/drivers/net/ixgbe/ixgbe_ethdev.c @@ -1631,8 +1631,8 @@ eth_ixgbevf_dev_init(struct rte_eth_dev *eth_dev) PMD_INIT_LOG(ERR, "VF Initialization Failure: %d", diag); /* * This error code will be propagated to the app by - * rte_eth_dev_reset, so use a public error code rather than - * the internal-only IXGBE_ERR_RESET_FAILED + * rte_eth_dev_reset_async, so use a public error code rather + * than the internal-only IXGBE_ERR_RESET_FAILED */ return -EAGAIN; } diff --git a/drivers/net/netvsc/hn_var.h b/drivers/net/netvsc/hn_var.h index 7f3266c45..5344f75c1 100644 --- a/drivers/net/netvsc/hn_var.h +++ b/drivers/net/netvsc/hn_var.h @@ -177,7 +177,7 @@ int hn_vf_configure(struct rte_eth_dev *dev, const struct rte_eth_conf *dev_conf); const uint32_t *hn_vf_supported_ptypes(struct rte_eth_dev *dev); int hn_vf_start(struct rte_eth_dev *dev); -void hn_vf_reset(struct rte_eth_dev *dev); +void hn_vf_reset_async(struct rte_eth_dev *dev); void hn_vf_stop(struct rte_eth_dev *dev); void hn_vf_close(struct rte_eth_dev *dev); diff --git a/drivers/net/netvsc/hn_vf.c b/drivers/net/netvsc/hn_vf.c index 3f714ec99..ae03455ba 100644 --- a/drivers/net/netvsc/hn_vf.c +++ b/drivers/net/netvsc/hn_vf.c @@ -358,9 +358,9 @@ void hn_vf_stop(struct rte_eth_dev *dev) rte_spinlock_unlock(&hv->vf_lock); \ } -void hn_vf_reset(struct rte_eth_dev *dev) +void hn_vf_reset_async(struct rte_eth_dev *dev) { - VF_ETHDEV_FUNC(dev, rte_eth_dev_reset); + VF_ETHDEV_FUNC(dev, rte_eth_dev_reset_async); } void hn_vf_close(struct rte_eth_dev *dev) diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c index 85c179496..5a6be6a5b 100644 --- a/lib/librte_ethdev/rte_ethdev.c +++ b/lib/librte_ethdev/rte_ethdev.c @@ -331,7 +331,8 @@ rte_eth_find_next(uint16_t port_id) { while (port_id < RTE_MAX_ETHPORTS && rte_eth_devices[port_id].state != RTE_ETH_DEV_ATTACHED && - rte_eth_devices[port_id].state != RTE_ETH_DEV_REMOVED) + rte_eth_devices[port_id].state != RTE_ETH_DEV_REMOVED && + rte_eth_devices[port_id].state != RTE_ETH_DEV_RESETTING) port_id++; if (port_id >= RTE_MAX_ETHPORTS) @@ -556,7 +557,8 @@ rte_eth_find_next_owned_by(uint16_t port_id, const uint64_t owner_id) { while (port_id < RTE_MAX_ETHPORTS && ((rte_eth_devices[port_id].state != RTE_ETH_DEV_ATTACHED && - rte_eth_devices[port_id].state != RTE_ETH_DEV_REMOVED) || + rte_eth_devices[port_id].state != RTE_ETH_DEV_REMOVED && + rte_eth_devices[port_id].state != RTE_ETH_DEV_RESETTING) || rte_eth_devices[port_id].data->owner.id != owner_id)) port_id++; @@ -1490,10 +1492,38 @@ rte_eth_dev_close(uint16_t port_id) dev->data->tx_queues = NULL; } +struct dev_reset_args { + struct rte_eth_dev *dev; + enum rte_eth_dev_state pre_state; +}; + +static void * +do_dev_reset(void *args) +{ + struct dev_reset_args *reset_args = args; + struct rte_eth_dev *dev = reset_args->dev; + int ret; + + ret = dev->dev_ops->dev_reset(dev); + + /*restore previous device state */ + dev->state = reset_args->pre_state; + + /* notify users */ + _rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_RESET_COMPLETE, + &ret); + + free(args); + return NULL; +} + int -rte_eth_dev_reset(uint16_t port_id) +rte_eth_dev_reset_async(uint16_t port_id) { struct rte_eth_dev *dev; + struct dev_reset_args *args; + char pthread_name[20]; + pthread_t tid; int ret; RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -EINVAL); @@ -1501,10 +1531,26 @@ rte_eth_dev_reset(uint16_t port_id) RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP); + /* already on resetting */ + if (dev->state == RTE_ETH_DEV_RESETTING) + return 0; + + args = calloc(1, sizeof(struct dev_reset_args)); + if (!args) + return -ENOMEM; + rte_eth_dev_stop(port_id); - ret = dev->dev_ops->dev_reset(dev); - return eth_err(port_id, ret); + /* store previous device state temporary */ + args->pre_state = dev->state; + + dev->state = RTE_ETH_DEV_RESETTING; + snprintf(pthread_name, sizeof(pthread_name), + "do_dev_reset_%d", port_id); + ret = rte_ctrl_thread_create(&tid, pthread_name, NULL, + do_dev_reset, args); + + return ret; } int __rte_experimental diff --git a/lib/librte_ethdev/rte_ethdev.h b/lib/librte_ethdev/rte_ethdev.h index a3c864a13..b02c16ae3 100644 --- a/lib/librte_ethdev/rte_ethdev.h +++ b/lib/librte_ethdev/rte_ethdev.h @@ -1308,6 +1308,8 @@ enum rte_eth_dev_state { RTE_ETH_DEV_ATTACHED, /** Device is in removed state when plug-out is detected. */ RTE_ETH_DEV_REMOVED, + /** Device is on resetting when rte_eth_dev_reset_async is called. */ + RTE_ETH_DEV_RESETTING, }; struct rte_eth_dev_sriov { @@ -1857,21 +1859,31 @@ void rte_eth_dev_close(uint16_t port_id); * RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it to start * a port reset in other circumstances. * - * When this function is called, it first stops the port and then calls the - * PMD specific dev_uninit( ) and dev_init( ) to return the port to initial - * state, in which no Tx and Rx queues are setup, as if the port has been - * reset and not started. The port keeps the port id it had before the - * function call. - * - * After calling rte_eth_dev_reset( ), the application should use - * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ), - * rte_eth_tx_queue_setup( ), and rte_eth_dev_start( ) - * to reconfigure the device as appropriate. - * - * Note: To avoid unexpected behavior, the application should stop calling - * Tx and Rx functions before calling rte_eth_dev_reset( ). For thread - * safety, all these controlling functions should be called from the same - * thread. + * @note + * Device reset may have the dependency, for example, a VF reset expects + * PF ready, or a NIC function as a part of a SOC need to wait for other + * parts of the system be ready, these are time-consuming tasks and will + * block current thread. + * + * As the name, rte_eth_dev_reset_async is an async API, it will spwan a + * new thread to call ops->dev_reset, once it is finished, it will raise + * the RTE_ETH_EVENT_RESET_COMPLETE event to notify application. That makes + * things easy for an application that want to reset the device from the + * interrupt thread since typically a RTE_ETH_EVENT_INTR_RESET handler is + * invoked in interrupt thread. The typical implementation of ops->dev_reset + * will do some hardware reset operations through calling dev_uninit() and + * dev_init(). + * + * Application should not assume device reset is finished after + * rte_eth_dev_reset_async return, it should always wait for a + * RTE_ETH_EVENT_RESET_COMPLETE event and check the reset result. + * If reset success, application should call rte_eth_dev_configure( ), + * rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ), + * and rte_eth_dev_start( ) to reconfigure the device as appropriate. + * + * @Note + * To avoid unexpected behavior, the application should stop calling + * Tx and Rx functions before calling rte_eth_dev_reset_async( ). * * @param port_id * The port identifier of the Ethernet device. @@ -1880,12 +1892,10 @@ void rte_eth_dev_close(uint16_t port_id); * - (0) if successful. * - (-EINVAL) if port identifier is invalid. * - (-ENOTSUP) if hardware doesn't support this function. - * - (-EPERM) if not ran from the primary process. - * - (-EIO) if re-initialisation failed or device is removed. * - (-ENOMEM) if the reset failed due to OOM. - * - (-EAGAIN) if the reset temporarily failed and should be retried later. + * - (<0) other errors from low level driver. */ -int rte_eth_dev_reset(uint16_t port_id); +int rte_eth_dev_reset_async(uint16_t port_id); /** * Enable receipt in promiscuous mode for an Ethernet device. @@ -2617,6 +2627,8 @@ enum rte_eth_event_type { /**< queue state event (enabled/disabled) */ RTE_ETH_EVENT_INTR_RESET, /**< reset interrupt event, sent to VF on PF reset */ + RTE_ETH_EVENT_RESET_COMPLETE, + /**< inform application that reset is completed */ RTE_ETH_EVENT_VF_MBOX, /**< message from the VF received by PF */ RTE_ETH_EVENT_MACSEC, /**< MACsec offload related event */ RTE_ETH_EVENT_INTR_RMV, /**< device removal event */ diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map index 92ac3de25..e9f02656a 100644 --- a/lib/librte_ethdev/rte_ethdev_version.map +++ b/lib/librte_ethdev/rte_ethdev_version.map @@ -229,6 +229,13 @@ DPDK_18.11 { } DPDK_18.08; +DPDK_19.05 { + global: + + rte_eth_dev_reset_async; + +} DPDK_18.08; + EXPERIMENTAL { global: -- 2.13.6