From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 27DED201 for ; Tue, 19 Mar 2019 14:13:34 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Mar 2019 06:13:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,498,1544515200"; d="scan'208";a="308483703" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.46]) ([10.237.221.46]) by orsmga005.jf.intel.com with ESMTP; 19 Mar 2019 06:13:32 -0700 To: "Zhang, Qi Z" , "thomas@monjalon.net" , "Doherty, Declan" Cc: "ktraynor@redhat.com" , "dev@dpdk.org" , "Shelton, Benjamin H" , "Vangati, Narender" References: <20180920045626.106109-1-qi.z.zhang@intel.com> <039ED4275CED7440929022BC67E70611532AA4B1@SHSMSX103.ccr.corp.intel.com> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= mQINBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABtCVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+iQJVBBMBAgA/AhsDBgsJCAcDAgYVCAIJCgsE FgIDAQIeAQIXgBYhBNI2U4dCLsKE45mBx/kz60PfE2EfBQJbughWBQkHwjOGAAoJEPkz60Pf E2Eft84QAIbKWqhgqRfoiw/BbXbA1+qm2o4UgkCRQ0yJgt9QsnbpOmPKydHH0ixCliNz1J8e mRXCkMini1bTpnzp7spOjQGLeAFkNFz6BMq8YF2mVWbGEDE9WgnAxZdi0eLY7ZQnHbE6AxKL SXmpe9INb6z3ztseFt7mqje/W/6DWYIMnH3Yz9KzxujFWDcq8UCAvPkxVQXLTMpauhFgYeEx Nub5HbvhxTfUkapLwRQsSd/HbywzqZ3s/bbYMjj5JO3tgMiM9g9HOjv1G2f1dQjHi5YQiTZl 1eIIqQ3pTic6ROaiZqNmQFXPsoOOFfXF8nN2zg8kl/sSdoXWHhama5hbwwtl1vdaygQYlmdK H2ueiFh/UvT3WG3waNv2eZiEbHV8Rk52Xyn2w1G90lV0fYC6Ket1Xjoch7kjwbx793Kz/RfQ rmBY8/S4DTGn3oq3dMdQY+b6+7VMUeLMMh2CXYO9ErkOq+qNTD1IY+cBAkXnaDbQfz0zbste ZGWH74FAZ9nCpDOqbRTrBL42aMGhfOWEyeA1x7+hl6JZfabBWAuf4nnCXuorKHzBXTrf7u7p fXsKQClWRW77PF1VmzrtKNVSytQAmlCWApQIw20AarFipXmVdIjHmJPU611WoyxZPb4JTOxx 5cv9B+nr/RIB+v5dcStyHCCwO1be7nBDdCgd4F6kTQPLuQINBFfWTL4BEACnNA29e8TarUsB L5n6eLZHXcFvVwNLVlirWOClHXf44o2KnN3ww+eBEmKVfEFo9MSuGDNHS8Zw1NiGMYxLIUgd U6gGrVVs/VrQWL82pbMk6jCj98N+BXIri+6K1z+AImz7ax7iF1kDgRAnFWU0znWWBgM2mM8Y gDjcxfXk4sCKnvf6Gjo08Ey5zmqx7dekAKU2EEp8Q1EJY3jbymLdZWRP4AFFMTS1rGMk0/tt v71NBg1GobCcbNfn9chK/jhqxYhAJqq86RdJQkt3/9x1U1Oq0vXCt4JVVHmkxePtUiuWTTt+ aYlUAsKYZsWvncExvw77x2ArYDmaK0yfjh37wp0lY7DOJHFxoyT8tyWZlLci/VMRG2Ja33xj 0CN4C1yBg+QDeV3QFxQo42iA/ykdXPUR3ezmsND3XKvVLTC4DNb3V/EZQ7jBj64+bEK0VW4G B31VP00ApNQvSoczsIOAKdk97RNbpmPw6q10ILIB+9T1xbnFYzshzGF17oC0/GENIHATx8vZ masOZoDiOZQpeneLgnFE9JfzhLTxv6wNZcc/HLXRQVTkDsQr8ERtkAoHCf1E5+b5Yr7pfnE4 YuhET746o25S53ELUYPIs49qoJsEJL34/oexMfPGyPIlrbufiNyty5jc/1MRwUlhJlJ5IOHy ZUa+6CLR7GdImusFkPJUJwARAQABiQI8BBgBAgAmAhsMFiEE0jZTh0IuwoTjmYHH+TPrQ98T YR8FAlu6CHAFCQXE7zIACgkQ+TPrQ98TYR9nXxAAqNBgkYNyGuWUuy0GwDQCbu3iiMyH1+D7 llafPcK4NYy1Z4AYuVwC9nmLaoj+ozdqS3ncRo57ncRsKEJC46nDJJZYZ5LSJVn63Y3NBF86 lxQAgjj2oyZEwaLKtKbAFsXL43jv1pUGgSvWwYtDwHITXXFQto9rZEuUDRFSx4sg9OR+Q6/6 LY+nQQ3OdHlBkflzYMPcWgDcvcTAO6yasLEUf7UcYoSWTyMYjLB4QuNlXzTswzGVMssJF/vo V8lD1eqqaSUWG3STF6GVLQOr1NLvN5+kUBiEStHFxBpgSCvYY9sNV8FS6N24CAWMBl+10W+D 2h1yiiP5dOdPcBDYKsgqDD91/sP0WdyMJkwdQJtD49f9f+lYloxHnSAxMleOpyscg1pldw+i mPaUY1bmIknLhhkqfMmjywQOXpac5LRMibAAYkcB8v7y3kwELnt8mhqqZy6LUsqcWygNbH/W K3GGt5tRpeIXeJ25x8gg5EBQ0Jnvp/IbBYQfPLtXH0Myq2QuAhk/1q2yEIbVjS+7iowEZNyE 56K63WBJxsJPB2mvmLgn98GqB4G6GufP1ndS0XDti/2K0o8rep9xoY/JDGi0n0L0tk9BHyoP Y7kaEpu7UyY3nVdRLe5H1/MnFG8hdJ97WqnPS0buYZlrbTV0nRFL/NI2VABl18vEEXvNQiO+ vM8= Message-ID: <4c60c418-e653-41ec-ef09-b348dbd7d140@intel.com> Date: Tue, 19 Mar 2019 13:13:31 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: <039ED4275CED7440929022BC67E70611532AA4B1@SHSMSX103.ccr.corp.intel.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [RFC v3] ethdev: claim device reset as async X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2019 13:13:35 -0000 On 10/4/2018 4:58 PM, Zhang, Qi Z wrote: > > >> -----Original Message----- >> From: Yigit, Ferruh >> Sent: Thursday, October 4, 2018 7:30 PM >> To: Zhang, Qi Z ; thomas@monjalon.net; Doherty, >> Declan >> Cc: ktraynor@redhat.com; dev@dpdk.org; Shelton, Benjamin H >> ; Vangati, Narender >> >> Subject: Re: [RFC v3] ethdev: claim device reset as async >> >> On 9/20/2018 5:56 AM, Qi Zhang wrote: >>> Device reset should be implemented in an async way since it is >>> possible to be invoked in interrupt thread and sometimes to reset a >>> device need to wait for some dependency, for example, a VF expects for >>> PF ready or a NIC function as part of a SOC wait for the whole system >>> reset complete, and all these time-consuming tasks will block the >>> interrupt thread. >>> The patch rename rte_eth_dev_reset to rte_eth_dev_reset_async and >>> rework the implementation. It will spawn a new thread which will call >>> ops->dev_reset, and when finished it will raise the event >>> RTE_ETH_EVENT_RESET_COMPLETE. The application should always wait for >>> this event before it continues to configure and restart the device. >>> >>> Signed-off-by: Qi Zhang >> >> <...> >> >>> @@ -1385,10 +1413,26 @@ rte_eth_dev_reset(uint16_t port_id) >>> >>> RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP); >>> >>> + /* already on resetting */ >>> + if (dev->state == RTE_ETH_DEV_RESETTING) >>> + return 0; >>> + >>> + args = calloc(1, sizeof(struct dev_reset_args)); >>> + if (!args) >>> + return -ENOMEM; >>> + >>> rte_eth_dev_stop(port_id); >>> - ret = dev->dev_ops->dev_reset(dev); >>> >>> - return eth_err(port_id, ret); >>> + /* store previous device state temporary */ >>> + args->pre_state = dev->state; >>> + >>> + dev->state = RTE_ETH_DEV_RESETTING; >> >> Do we need to update the state, I think this will break rte_eth_dev_count() and >> friends, like during device reset app will think it has one less device in system. > > I'd like to have this new state which represent the situation of the device more accurate. > In this patch RTE_ETH_DEV_RESETTING is just to be used to prevent double reset, but in future it can also be used to prevent invalid operation during device reset. > > Of cause we need to make sure it does not break exist behavior and seems add RTE_ETH_DEV_RESETTING check in rte_eth_find_next_owned_by and rte_eth_find_next is able to fix the issue you observed. > > I can add this in v4 if you agree the idea. > >> >> <...> >> >>> @@ -1814,21 +1816,29 @@ void rte_eth_dev_close(uint16_t port_id); >>> * RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it to >> start >>> * a port reset in other circumstances. >>> * >>> - * When this function is called, it first stops the port and then >>> calls the >>> - * PMD specific dev_uninit( ) and dev_init( ) to return the port to >>> initial >>> - * state, in which no Tx and Rx queues are setup, as if the port has >>> been >>> - * reset and not started. The port keeps the port id it had before >>> the >>> - * function call. >>> - * >>> - * After calling rte_eth_dev_reset( ), the application should use >>> - * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ), >>> - * rte_eth_tx_queue_setup( ), and rte_eth_dev_start( ) >>> - * to reconfigure the device as appropriate. >>> - * >>> - * Note: To avoid unexpected behavior, the application should stop >>> calling >>> - * Tx and Rx functions before calling rte_eth_dev_reset( ). For >>> thread >>> - * safety, all these controlling functions should be called from the >>> same >>> - * thread. >>> + * @note >>> + * Device reset may have the dependency, for example, a VF reset >>> + expects >>> + * PF ready, or a NIC function as a part of a SOC need to wait for >>> + other >>> + * parts of the system be ready, these are time-consuming tasks and >>> + will >>> + * block current thread. >>> + * >>> + * As the name, rte_eth_dev_reset_async is an async API, it will >>> + spwan a >>> + * new thread to call ops->dev_reset, once it is finished, it will >>> + raise >>> + * the RTE_ETH_EVENT_RESET_COMPLETE event to notify application. >>> + That makes >>> + * things easy for an application that what to reset the device from >>> + the >>> + * interrupt thread since typically a RTE_ETH_EVENT_INTR_RESET >>> + handler is >>> + * invoked in interrupt thread. >> >> thread calls dev_ops->dev_reset(dev) and wait for it, so it means >> dev_ops->dev_reset is synchronous, perhaps it would be good to highlight this >> in "dev_reset" comment to help PMD developers. > > OK > >> >> of dev_ops->dev_reset() is synchronous, means existing rte_eth_dev_reset() is >> synchronous, so what do you thinks keep rte_eth_dev_reset() as it is and add >> new >> rte_eth_dev_reset_async() API? Than we will have both sync and async >> solution. > > Typically device reset happens when application receive RTE_ETH_EVENT_INTR_RESET and this is in interrupt thread. > Invoke an async API in interrupt thread is the right way, is it better if we highlight this is the only way? > I may not prefer to expose the sync API right now, it's better to figure out some typical usage before we expose this, but so far I don't have. Hi Qi, Is the 'rte_eth_dev_reset_async()' still required? Is there any update planned to this RFC? > > Regards > Qi > > >> >>> + * >>> + * Application should not assume device reset is finished after >>> + * rte_eth_dev_reset_async return, it should always wait for a >>> + * RTE_ETH_EVENT_RESET_COMPLETE event and check the reset result. >>> + * If reset success, application should call rte_eth_dev_configure( >>> + ), >>> + * rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ), >>> + * and rte_eth_dev_start( ) to reconfigure the device as appropriate. >>> + * >>> + * @Note >>> + * To avoid unexpected behavior, the application should stop calling >>> + * Tx and Rx functions before calling rte_eth_dev_reset_async( ). >>> * >>> * @param port_id >>> * The port identifier of the Ethernet device. >>> @@ -1837,12 +1847,10 @@ void rte_eth_dev_close(uint16_t port_id); >>> * - (0) if successful. >>> * - (-EINVAL) if port identifier is invalid. >>> * - (-ENOTSUP) if hardware doesn't support this function. >>> - * - (-EPERM) if not ran from the primary process. >>> - * - (-EIO) if re-initialisation failed or device is removed. >>> * - (-ENOMEM) if the reset failed due to OOM. >>> - * - (-EAGAIN) if the reset temporarily failed and should be retried later. >>> + * - (<0) other errors from low level driver. >>> */ >>> -int rte_eth_dev_reset(uint16_t port_id); >>> +int rte_eth_dev_reset_async(uint16_t port_id); >>> >>> /** >>> * Enable receipt in promiscuous mode for an Ethernet device. >>> @@ -2574,6 +2582,8 @@ enum rte_eth_event_type { >>> /**< queue state event (enabled/disabled) */ >>> RTE_ETH_EVENT_INTR_RESET, >>> /**< reset interrupt event, sent to VF on PF reset */ >>> + RTE_ETH_EVENT_RESET_COMPLETE, >>> + /**< inform application that reset is completed */ >>> RTE_ETH_EVENT_VF_MBOX, /**< message from the VF received by PF >> */ >>> RTE_ETH_EVENT_MACSEC, /**< MACsec offload related event */ >>> RTE_ETH_EVENT_INTR_RMV, /**< device removal event */ >>> > From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by dpdk.space (Postfix) with ESMTP id 53944A00E6 for ; Tue, 19 Mar 2019 14:13:38 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1C49011A4; Tue, 19 Mar 2019 14:13:37 +0100 (CET) Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 27DED201 for ; Tue, 19 Mar 2019 14:13:34 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Mar 2019 06:13:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.58,498,1544515200"; d="scan'208";a="308483703" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.46]) ([10.237.221.46]) by orsmga005.jf.intel.com with ESMTP; 19 Mar 2019 06:13:32 -0700 To: "Zhang, Qi Z" , "thomas@monjalon.net" , "Doherty, Declan" Cc: "ktraynor@redhat.com" , "dev@dpdk.org" , "Shelton, Benjamin H" , "Vangati, Narender" References: <20180920045626.106109-1-qi.z.zhang@intel.com> <039ED4275CED7440929022BC67E70611532AA4B1@SHSMSX103.ccr.corp.intel.com> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= mQINBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABtCVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+iQJVBBMBAgA/AhsDBgsJCAcDAgYVCAIJCgsE FgIDAQIeAQIXgBYhBNI2U4dCLsKE45mBx/kz60PfE2EfBQJbughWBQkHwjOGAAoJEPkz60Pf E2Eft84QAIbKWqhgqRfoiw/BbXbA1+qm2o4UgkCRQ0yJgt9QsnbpOmPKydHH0ixCliNz1J8e mRXCkMini1bTpnzp7spOjQGLeAFkNFz6BMq8YF2mVWbGEDE9WgnAxZdi0eLY7ZQnHbE6AxKL SXmpe9INb6z3ztseFt7mqje/W/6DWYIMnH3Yz9KzxujFWDcq8UCAvPkxVQXLTMpauhFgYeEx Nub5HbvhxTfUkapLwRQsSd/HbywzqZ3s/bbYMjj5JO3tgMiM9g9HOjv1G2f1dQjHi5YQiTZl 1eIIqQ3pTic6ROaiZqNmQFXPsoOOFfXF8nN2zg8kl/sSdoXWHhama5hbwwtl1vdaygQYlmdK H2ueiFh/UvT3WG3waNv2eZiEbHV8Rk52Xyn2w1G90lV0fYC6Ket1Xjoch7kjwbx793Kz/RfQ rmBY8/S4DTGn3oq3dMdQY+b6+7VMUeLMMh2CXYO9ErkOq+qNTD1IY+cBAkXnaDbQfz0zbste ZGWH74FAZ9nCpDOqbRTrBL42aMGhfOWEyeA1x7+hl6JZfabBWAuf4nnCXuorKHzBXTrf7u7p fXsKQClWRW77PF1VmzrtKNVSytQAmlCWApQIw20AarFipXmVdIjHmJPU611WoyxZPb4JTOxx 5cv9B+nr/RIB+v5dcStyHCCwO1be7nBDdCgd4F6kTQPLuQINBFfWTL4BEACnNA29e8TarUsB L5n6eLZHXcFvVwNLVlirWOClHXf44o2KnN3ww+eBEmKVfEFo9MSuGDNHS8Zw1NiGMYxLIUgd U6gGrVVs/VrQWL82pbMk6jCj98N+BXIri+6K1z+AImz7ax7iF1kDgRAnFWU0znWWBgM2mM8Y gDjcxfXk4sCKnvf6Gjo08Ey5zmqx7dekAKU2EEp8Q1EJY3jbymLdZWRP4AFFMTS1rGMk0/tt v71NBg1GobCcbNfn9chK/jhqxYhAJqq86RdJQkt3/9x1U1Oq0vXCt4JVVHmkxePtUiuWTTt+ aYlUAsKYZsWvncExvw77x2ArYDmaK0yfjh37wp0lY7DOJHFxoyT8tyWZlLci/VMRG2Ja33xj 0CN4C1yBg+QDeV3QFxQo42iA/ykdXPUR3ezmsND3XKvVLTC4DNb3V/EZQ7jBj64+bEK0VW4G B31VP00ApNQvSoczsIOAKdk97RNbpmPw6q10ILIB+9T1xbnFYzshzGF17oC0/GENIHATx8vZ masOZoDiOZQpeneLgnFE9JfzhLTxv6wNZcc/HLXRQVTkDsQr8ERtkAoHCf1E5+b5Yr7pfnE4 YuhET746o25S53ELUYPIs49qoJsEJL34/oexMfPGyPIlrbufiNyty5jc/1MRwUlhJlJ5IOHy ZUa+6CLR7GdImusFkPJUJwARAQABiQI8BBgBAgAmAhsMFiEE0jZTh0IuwoTjmYHH+TPrQ98T YR8FAlu6CHAFCQXE7zIACgkQ+TPrQ98TYR9nXxAAqNBgkYNyGuWUuy0GwDQCbu3iiMyH1+D7 llafPcK4NYy1Z4AYuVwC9nmLaoj+ozdqS3ncRo57ncRsKEJC46nDJJZYZ5LSJVn63Y3NBF86 lxQAgjj2oyZEwaLKtKbAFsXL43jv1pUGgSvWwYtDwHITXXFQto9rZEuUDRFSx4sg9OR+Q6/6 LY+nQQ3OdHlBkflzYMPcWgDcvcTAO6yasLEUf7UcYoSWTyMYjLB4QuNlXzTswzGVMssJF/vo V8lD1eqqaSUWG3STF6GVLQOr1NLvN5+kUBiEStHFxBpgSCvYY9sNV8FS6N24CAWMBl+10W+D 2h1yiiP5dOdPcBDYKsgqDD91/sP0WdyMJkwdQJtD49f9f+lYloxHnSAxMleOpyscg1pldw+i mPaUY1bmIknLhhkqfMmjywQOXpac5LRMibAAYkcB8v7y3kwELnt8mhqqZy6LUsqcWygNbH/W K3GGt5tRpeIXeJ25x8gg5EBQ0Jnvp/IbBYQfPLtXH0Myq2QuAhk/1q2yEIbVjS+7iowEZNyE 56K63WBJxsJPB2mvmLgn98GqB4G6GufP1ndS0XDti/2K0o8rep9xoY/JDGi0n0L0tk9BHyoP Y7kaEpu7UyY3nVdRLe5H1/MnFG8hdJ97WqnPS0buYZlrbTV0nRFL/NI2VABl18vEEXvNQiO+ vM8= Message-ID: <4c60c418-e653-41ec-ef09-b348dbd7d140@intel.com> Date: Tue, 19 Mar 2019 13:13:31 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.5.3 MIME-Version: 1.0 In-Reply-To: <039ED4275CED7440929022BC67E70611532AA4B1@SHSMSX103.ccr.corp.intel.com> Content-Type: text/plain; charset="UTF-8" Content-Language: en-US Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [RFC v3] ethdev: claim device reset as async X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Message-ID: <20190319131331.jvzt3rhpXkVNKN__Gf89AfXF_jgbGgJKug5xy-QI6cU@z> On 10/4/2018 4:58 PM, Zhang, Qi Z wrote: > > >> -----Original Message----- >> From: Yigit, Ferruh >> Sent: Thursday, October 4, 2018 7:30 PM >> To: Zhang, Qi Z ; thomas@monjalon.net; Doherty, >> Declan >> Cc: ktraynor@redhat.com; dev@dpdk.org; Shelton, Benjamin H >> ; Vangati, Narender >> >> Subject: Re: [RFC v3] ethdev: claim device reset as async >> >> On 9/20/2018 5:56 AM, Qi Zhang wrote: >>> Device reset should be implemented in an async way since it is >>> possible to be invoked in interrupt thread and sometimes to reset a >>> device need to wait for some dependency, for example, a VF expects for >>> PF ready or a NIC function as part of a SOC wait for the whole system >>> reset complete, and all these time-consuming tasks will block the >>> interrupt thread. >>> The patch rename rte_eth_dev_reset to rte_eth_dev_reset_async and >>> rework the implementation. It will spawn a new thread which will call >>> ops->dev_reset, and when finished it will raise the event >>> RTE_ETH_EVENT_RESET_COMPLETE. The application should always wait for >>> this event before it continues to configure and restart the device. >>> >>> Signed-off-by: Qi Zhang >> >> <...> >> >>> @@ -1385,10 +1413,26 @@ rte_eth_dev_reset(uint16_t port_id) >>> >>> RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset, -ENOTSUP); >>> >>> + /* already on resetting */ >>> + if (dev->state == RTE_ETH_DEV_RESETTING) >>> + return 0; >>> + >>> + args = calloc(1, sizeof(struct dev_reset_args)); >>> + if (!args) >>> + return -ENOMEM; >>> + >>> rte_eth_dev_stop(port_id); >>> - ret = dev->dev_ops->dev_reset(dev); >>> >>> - return eth_err(port_id, ret); >>> + /* store previous device state temporary */ >>> + args->pre_state = dev->state; >>> + >>> + dev->state = RTE_ETH_DEV_RESETTING; >> >> Do we need to update the state, I think this will break rte_eth_dev_count() and >> friends, like during device reset app will think it has one less device in system. > > I'd like to have this new state which represent the situation of the device more accurate. > In this patch RTE_ETH_DEV_RESETTING is just to be used to prevent double reset, but in future it can also be used to prevent invalid operation during device reset. > > Of cause we need to make sure it does not break exist behavior and seems add RTE_ETH_DEV_RESETTING check in rte_eth_find_next_owned_by and rte_eth_find_next is able to fix the issue you observed. > > I can add this in v4 if you agree the idea. > >> >> <...> >> >>> @@ -1814,21 +1816,29 @@ void rte_eth_dev_close(uint16_t port_id); >>> * RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it to >> start >>> * a port reset in other circumstances. >>> * >>> - * When this function is called, it first stops the port and then >>> calls the >>> - * PMD specific dev_uninit( ) and dev_init( ) to return the port to >>> initial >>> - * state, in which no Tx and Rx queues are setup, as if the port has >>> been >>> - * reset and not started. The port keeps the port id it had before >>> the >>> - * function call. >>> - * >>> - * After calling rte_eth_dev_reset( ), the application should use >>> - * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ), >>> - * rte_eth_tx_queue_setup( ), and rte_eth_dev_start( ) >>> - * to reconfigure the device as appropriate. >>> - * >>> - * Note: To avoid unexpected behavior, the application should stop >>> calling >>> - * Tx and Rx functions before calling rte_eth_dev_reset( ). For >>> thread >>> - * safety, all these controlling functions should be called from the >>> same >>> - * thread. >>> + * @note >>> + * Device reset may have the dependency, for example, a VF reset >>> + expects >>> + * PF ready, or a NIC function as a part of a SOC need to wait for >>> + other >>> + * parts of the system be ready, these are time-consuming tasks and >>> + will >>> + * block current thread. >>> + * >>> + * As the name, rte_eth_dev_reset_async is an async API, it will >>> + spwan a >>> + * new thread to call ops->dev_reset, once it is finished, it will >>> + raise >>> + * the RTE_ETH_EVENT_RESET_COMPLETE event to notify application. >>> + That makes >>> + * things easy for an application that what to reset the device from >>> + the >>> + * interrupt thread since typically a RTE_ETH_EVENT_INTR_RESET >>> + handler is >>> + * invoked in interrupt thread. >> >> thread calls dev_ops->dev_reset(dev) and wait for it, so it means >> dev_ops->dev_reset is synchronous, perhaps it would be good to highlight this >> in "dev_reset" comment to help PMD developers. > > OK > >> >> of dev_ops->dev_reset() is synchronous, means existing rte_eth_dev_reset() is >> synchronous, so what do you thinks keep rte_eth_dev_reset() as it is and add >> new >> rte_eth_dev_reset_async() API? Than we will have both sync and async >> solution. > > Typically device reset happens when application receive RTE_ETH_EVENT_INTR_RESET and this is in interrupt thread. > Invoke an async API in interrupt thread is the right way, is it better if we highlight this is the only way? > I may not prefer to expose the sync API right now, it's better to figure out some typical usage before we expose this, but so far I don't have. Hi Qi, Is the 'rte_eth_dev_reset_async()' still required? Is there any update planned to this RFC? > > Regards > Qi > > >> >>> + * >>> + * Application should not assume device reset is finished after >>> + * rte_eth_dev_reset_async return, it should always wait for a >>> + * RTE_ETH_EVENT_RESET_COMPLETE event and check the reset result. >>> + * If reset success, application should call rte_eth_dev_configure( >>> + ), >>> + * rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ), >>> + * and rte_eth_dev_start( ) to reconfigure the device as appropriate. >>> + * >>> + * @Note >>> + * To avoid unexpected behavior, the application should stop calling >>> + * Tx and Rx functions before calling rte_eth_dev_reset_async( ). >>> * >>> * @param port_id >>> * The port identifier of the Ethernet device. >>> @@ -1837,12 +1847,10 @@ void rte_eth_dev_close(uint16_t port_id); >>> * - (0) if successful. >>> * - (-EINVAL) if port identifier is invalid. >>> * - (-ENOTSUP) if hardware doesn't support this function. >>> - * - (-EPERM) if not ran from the primary process. >>> - * - (-EIO) if re-initialisation failed or device is removed. >>> * - (-ENOMEM) if the reset failed due to OOM. >>> - * - (-EAGAIN) if the reset temporarily failed and should be retried later. >>> + * - (<0) other errors from low level driver. >>> */ >>> -int rte_eth_dev_reset(uint16_t port_id); >>> +int rte_eth_dev_reset_async(uint16_t port_id); >>> >>> /** >>> * Enable receipt in promiscuous mode for an Ethernet device. >>> @@ -2574,6 +2582,8 @@ enum rte_eth_event_type { >>> /**< queue state event (enabled/disabled) */ >>> RTE_ETH_EVENT_INTR_RESET, >>> /**< reset interrupt event, sent to VF on PF reset */ >>> + RTE_ETH_EVENT_RESET_COMPLETE, >>> + /**< inform application that reset is completed */ >>> RTE_ETH_EVENT_VF_MBOX, /**< message from the VF received by PF >> */ >>> RTE_ETH_EVENT_MACSEC, /**< MACsec offload related event */ >>> RTE_ETH_EVENT_INTR_RMV, /**< device removal event */ >>> >