From mboxrd@z Thu Jan  1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from dpdk.org (dpdk.org [92.243.14.124])
	by dpdk.space (Postfix) with ESMTP id 2C490A00E6
	for <public@inbox.dpdk.org>; Tue, 19 Mar 2019 17:42:01 +0100 (CET)
Received: from [92.243.14.124] (localhost [127.0.0.1])
	by dpdk.org (Postfix) with ESMTP id 166D925A1;
	Tue, 19 Mar 2019 17:42:00 +0100 (CET)
Received: from mga01.intel.com (mga01.intel.com [192.55.52.88])
 by dpdk.org (Postfix) with ESMTP id 327F51DBD
 for <dev@dpdk.org>; Tue, 19 Mar 2019 17:41:58 +0100 (CET)
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from orsmga005.jf.intel.com ([10.7.209.41])
 by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384;
 19 Mar 2019 09:41:57 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.58,498,1544515200"; d="scan'208";a="308537076"
Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.46])
 ([10.237.221.46])
 by orsmga005.jf.intel.com with ESMTP; 19 Mar 2019 09:41:54 -0700
To: "Zhang, Qi Z" <qi.z.zhang@intel.com>,
 "thomas@monjalon.net" <thomas@monjalon.net>,
 "Doherty, Declan" <declan.doherty@intel.com>
Cc: "ktraynor@redhat.com" <ktraynor@redhat.com>, "dev@dpdk.org"
 <dev@dpdk.org>, "Shelton, Benjamin H" <benjamin.h.shelton@intel.com>,
 "Vangati, Narender" <narender.vangati@intel.com>
References: <20180920045626.106109-1-qi.z.zhang@intel.com>
 <b4388875-2404-e33e-3242-9f5c76db7b70@intel.com>
 <039ED4275CED7440929022BC67E70611532AA4B1@SHSMSX103.ccr.corp.intel.com>
 <4c60c418-e653-41ec-ef09-b348dbd7d140@intel.com>
 <039ED4275CED7440929022BC67E706115334F196@SHSMSX103.ccr.corp.intel.com>
From: Ferruh Yigit <ferruh.yigit@intel.com>
Openpgp: preference=signencrypt
Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata=
 mQINBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy
 qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ
 +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9
 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb
 +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF
 YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy
 ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX
 CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1
 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz
 cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABtCVGZXJydWggWWln
 aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+iQJVBBMBAgA/AhsDBgsJCAcDAgYVCAIJCgsE
 FgIDAQIeAQIXgBYhBNI2U4dCLsKE45mBx/kz60PfE2EfBQJbughWBQkHwjOGAAoJEPkz60Pf
 E2Eft84QAIbKWqhgqRfoiw/BbXbA1+qm2o4UgkCRQ0yJgt9QsnbpOmPKydHH0ixCliNz1J8e
 mRXCkMini1bTpnzp7spOjQGLeAFkNFz6BMq8YF2mVWbGEDE9WgnAxZdi0eLY7ZQnHbE6AxKL
 SXmpe9INb6z3ztseFt7mqje/W/6DWYIMnH3Yz9KzxujFWDcq8UCAvPkxVQXLTMpauhFgYeEx
 Nub5HbvhxTfUkapLwRQsSd/HbywzqZ3s/bbYMjj5JO3tgMiM9g9HOjv1G2f1dQjHi5YQiTZl
 1eIIqQ3pTic6ROaiZqNmQFXPsoOOFfXF8nN2zg8kl/sSdoXWHhama5hbwwtl1vdaygQYlmdK
 H2ueiFh/UvT3WG3waNv2eZiEbHV8Rk52Xyn2w1G90lV0fYC6Ket1Xjoch7kjwbx793Kz/RfQ
 rmBY8/S4DTGn3oq3dMdQY+b6+7VMUeLMMh2CXYO9ErkOq+qNTD1IY+cBAkXnaDbQfz0zbste
 ZGWH74FAZ9nCpDOqbRTrBL42aMGhfOWEyeA1x7+hl6JZfabBWAuf4nnCXuorKHzBXTrf7u7p
 fXsKQClWRW77PF1VmzrtKNVSytQAmlCWApQIw20AarFipXmVdIjHmJPU611WoyxZPb4JTOxx
 5cv9B+nr/RIB+v5dcStyHCCwO1be7nBDdCgd4F6kTQPLuQINBFfWTL4BEACnNA29e8TarUsB
 L5n6eLZHXcFvVwNLVlirWOClHXf44o2KnN3ww+eBEmKVfEFo9MSuGDNHS8Zw1NiGMYxLIUgd
 U6gGrVVs/VrQWL82pbMk6jCj98N+BXIri+6K1z+AImz7ax7iF1kDgRAnFWU0znWWBgM2mM8Y
 gDjcxfXk4sCKnvf6Gjo08Ey5zmqx7dekAKU2EEp8Q1EJY3jbymLdZWRP4AFFMTS1rGMk0/tt
 v71NBg1GobCcbNfn9chK/jhqxYhAJqq86RdJQkt3/9x1U1Oq0vXCt4JVVHmkxePtUiuWTTt+
 aYlUAsKYZsWvncExvw77x2ArYDmaK0yfjh37wp0lY7DOJHFxoyT8tyWZlLci/VMRG2Ja33xj
 0CN4C1yBg+QDeV3QFxQo42iA/ykdXPUR3ezmsND3XKvVLTC4DNb3V/EZQ7jBj64+bEK0VW4G
 B31VP00ApNQvSoczsIOAKdk97RNbpmPw6q10ILIB+9T1xbnFYzshzGF17oC0/GENIHATx8vZ
 masOZoDiOZQpeneLgnFE9JfzhLTxv6wNZcc/HLXRQVTkDsQr8ERtkAoHCf1E5+b5Yr7pfnE4
 YuhET746o25S53ELUYPIs49qoJsEJL34/oexMfPGyPIlrbufiNyty5jc/1MRwUlhJlJ5IOHy
 ZUa+6CLR7GdImusFkPJUJwARAQABiQI8BBgBAgAmAhsMFiEE0jZTh0IuwoTjmYHH+TPrQ98T
 YR8FAlu6CHAFCQXE7zIACgkQ+TPrQ98TYR9nXxAAqNBgkYNyGuWUuy0GwDQCbu3iiMyH1+D7
 llafPcK4NYy1Z4AYuVwC9nmLaoj+ozdqS3ncRo57ncRsKEJC46nDJJZYZ5LSJVn63Y3NBF86
 lxQAgjj2oyZEwaLKtKbAFsXL43jv1pUGgSvWwYtDwHITXXFQto9rZEuUDRFSx4sg9OR+Q6/6
 LY+nQQ3OdHlBkflzYMPcWgDcvcTAO6yasLEUf7UcYoSWTyMYjLB4QuNlXzTswzGVMssJF/vo
 V8lD1eqqaSUWG3STF6GVLQOr1NLvN5+kUBiEStHFxBpgSCvYY9sNV8FS6N24CAWMBl+10W+D
 2h1yiiP5dOdPcBDYKsgqDD91/sP0WdyMJkwdQJtD49f9f+lYloxHnSAxMleOpyscg1pldw+i
 mPaUY1bmIknLhhkqfMmjywQOXpac5LRMibAAYkcB8v7y3kwELnt8mhqqZy6LUsqcWygNbH/W
 K3GGt5tRpeIXeJ25x8gg5EBQ0Jnvp/IbBYQfPLtXH0Myq2QuAhk/1q2yEIbVjS+7iowEZNyE
 56K63WBJxsJPB2mvmLgn98GqB4G6GufP1ndS0XDti/2K0o8rep9xoY/JDGi0n0L0tk9BHyoP
 Y7kaEpu7UyY3nVdRLe5H1/MnFG8hdJ97WqnPS0buYZlrbTV0nRFL/NI2VABl18vEEXvNQiO+ vM8=
Message-ID: <aeec17cd-b3c1-4c65-6d9a-a89ba9cf609e@intel.com>
Date: Tue, 19 Mar 2019 16:41:54 +0000
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101
 Thunderbird/60.5.3
MIME-Version: 1.0
In-Reply-To: <039ED4275CED7440929022BC67E706115334F196@SHSMSX103.ccr.corp.intel.com>
Content-Type: text/plain; charset="UTF-8"
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Subject: Re: [dpdk-dev] [RFC v3] ethdev: claim device reset as async
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
 <mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
 <mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
Message-ID: <20190319164154.0BnUF3nEjkG-ZqLyY182tEvrbTVv1BDgP8POB4xmCrc@z>

On 3/19/2019 1:40 PM, Zhang, Qi Z wrote:
> 
> 
>> -----Original Message-----
>> From: Yigit, Ferruh
>> Sent: Tuesday, March 19, 2019 9:14 PM
>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Doherty, Declan
>> <declan.doherty@intel.com>
>> Cc: ktraynor@redhat.com; dev@dpdk.org; Shelton, Benjamin H
>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>> <narender.vangati@intel.com>
>> Subject: Re: [RFC v3] ethdev: claim device reset as async
>>
>> On 10/4/2018 4:58 PM, Zhang, Qi Z wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Yigit, Ferruh
>>>> Sent: Thursday, October 4, 2018 7:30 PM
>>>> To: Zhang, Qi Z <qi.z.zhang@intel.com>; thomas@monjalon.net; Doherty,
>>>> Declan <declan.doherty@intel.com>
>>>> Cc: ktraynor@redhat.com; dev@dpdk.org; Shelton, Benjamin H
>>>> <benjamin.h.shelton@intel.com>; Vangati, Narender
>>>> <narender.vangati@intel.com>
>>>> Subject: Re: [RFC v3] ethdev: claim device reset as async
>>>>
>>>> On 9/20/2018 5:56 AM, Qi Zhang wrote:
>>>>> Device reset should be implemented in an async way since it is
>>>>> possible to be invoked in interrupt thread and sometimes to reset a
>>>>> device need to wait for some dependency, for example, a VF expects
>>>>> for PF ready or a NIC function as part of a SOC wait for the whole
>>>>> system reset complete, and all these time-consuming tasks will block
>>>>> the interrupt thread.
>>>>> The patch rename rte_eth_dev_reset to rte_eth_dev_reset_async and
>>>>> rework the implementation. It will spawn a new thread which will
>>>>> call
>>>>> ops->dev_reset, and when finished it will raise the event
>>>>> RTE_ETH_EVENT_RESET_COMPLETE. The application should always wait for
>>>>> this event before it continues to configure and restart the device.
>>>>>
>>>>> Signed-off-by: Qi Zhang <qi.z.zhang@intel.com>
>>>>
>>>> <...>
>>>>
>>>>> @@ -1385,10 +1413,26 @@ rte_eth_dev_reset(uint16_t port_id)
>>>>>
>>>>>  	RTE_FUNC_PTR_OR_ERR_RET(*dev->dev_ops->dev_reset,
>> -ENOTSUP);
>>>>>
>>>>> +	/* already on resetting */
>>>>> +	if (dev->state == RTE_ETH_DEV_RESETTING)
>>>>> +		return 0;
>>>>> +
>>>>> +	args = calloc(1, sizeof(struct dev_reset_args));
>>>>> +	if (!args)
>>>>> +		return -ENOMEM;
>>>>> +
>>>>>  	rte_eth_dev_stop(port_id);
>>>>> -	ret = dev->dev_ops->dev_reset(dev);
>>>>>
>>>>> -	return eth_err(port_id, ret);
>>>>> +	/* store previous device state temporary */
>>>>> +	args->pre_state = dev->state;
>>>>> +
>>>>> +	dev->state = RTE_ETH_DEV_RESETTING;
>>>>
>>>> Do we need to update the state, I think this will break
>>>> rte_eth_dev_count() and friends, like during device reset app will think it has
>> one less device in system.
>>>
>>> I'd like to have this new state which represent the situation of the device more
>> accurate.
>>> In this patch RTE_ETH_DEV_RESETTING is just to be used to prevent double
>> reset, but in future it can also be used to prevent invalid operation during device
>> reset.
>>>
>>> Of cause we need to make sure it does not break exist behavior and seems add
>> RTE_ETH_DEV_RESETTING check in rte_eth_find_next_owned_by and
>> rte_eth_find_next is able to fix the issue you observed.
>>>
>>> I can add this in v4 if you agree the idea.
>>>
>>>>
>>>> <...>
>>>>
>>>>> @@ -1814,21 +1816,29 @@ void rte_eth_dev_close(uint16_t port_id);
>>>>>   * RTE_ETH_EVENT_INTR_RESET event is detected, but can also use it
>>>>> to
>>>> start
>>>>>   * a port reset in other circumstances.
>>>>>   *
>>>>> - * When this function is called, it first stops the port and then
>>>>> calls the
>>>>> - * PMD specific dev_uninit( ) and dev_init( ) to return the port to
>>>>> initial
>>>>> - * state, in which no Tx and Rx queues are setup, as if the port
>>>>> has been
>>>>> - * reset and not started. The port keeps the port id it had before
>>>>> the
>>>>> - * function call.
>>>>> - *
>>>>> - * After calling rte_eth_dev_reset( ), the application should use
>>>>> - * rte_eth_dev_configure( ), rte_eth_rx_queue_setup( ),
>>>>> - * rte_eth_tx_queue_setup( ), and rte_eth_dev_start( )
>>>>> - * to reconfigure the device as appropriate.
>>>>> - *
>>>>> - * Note: To avoid unexpected behavior, the application should stop
>>>>> calling
>>>>> - * Tx and Rx functions before calling rte_eth_dev_reset( ). For
>>>>> thread
>>>>> - * safety, all these controlling functions should be called from
>>>>> the same
>>>>> - * thread.
>>>>> + * @note
>>>>> + * Device reset may have the dependency, for example, a VF reset
>>>>> + expects
>>>>> + * PF ready, or a NIC function as a part of a SOC need to wait for
>>>>> + other
>>>>> + * parts of the system be ready, these are time-consuming tasks and
>>>>> + will
>>>>> + * block current thread.
>>>>> + *
>>>>> + * As the name, rte_eth_dev_reset_async is an async API, it will
>>>>> + spwan a
>>>>> + * new thread to call ops->dev_reset, once it is finished, it will
>>>>> + raise
>>>>> + * the RTE_ETH_EVENT_RESET_COMPLETE event to notify application.
>>>>> + That makes
>>>>> + * things easy for an application that what to reset the device
>>>>> + from the
>>>>> + * interrupt thread since typically a RTE_ETH_EVENT_INTR_RESET
>>>>> + handler is
>>>>> + * invoked in interrupt thread.
>>>>
>>>> thread calls dev_ops->dev_reset(dev) and wait for it, so it means
>>>> dev_ops->dev_reset is synchronous, perhaps it would be good to
>>>> highlight this in "dev_reset" comment to help PMD developers.
>>>
>>> OK
>>>
>>>>
>>>> of dev_ops->dev_reset() is synchronous, means existing
>>>> rte_eth_dev_reset() is synchronous, so what do you thinks keep
>>>> rte_eth_dev_reset() as it is and add new
>>>> rte_eth_dev_reset_async() API? Than we will have both sync and async
>>>> solution.
>>>
>>> Typically device reset happens when application receive
>> RTE_ETH_EVENT_INTR_RESET and this is in interrupt thread.
>>> Invoke an async API in interrupt thread is the right way, is it better if we
>> highlight this is the only way?
>>> I may not prefer to expose the sync API right now, it's better to figure out some
>> typical usage before we expose this, but so far I don't have.
>>
>> Hi Qi,
>>
>> Is the 'rte_eth_dev_reset_async()' still required? Is there any update planned to
>> this RFC?
> 
> Yes, I think the requirement is still there. Just don't have bandwidth work on this recently.
> May I send out v1 for 19.05 in this week? since deprecation notes already be send out in 19.02 cycle

I think technically yes, since proposal is already out for a long time.
But it will give less time to review it when you send this week.

> 
> 
>>
>>>
>>> Regards
>>> Qi
>>>
>>>
>>>>
>>>>> + *
>>>>> + * Application should not assume device reset is finished after
>>>>> + * rte_eth_dev_reset_async return, it should always wait for a
>>>>> + * RTE_ETH_EVENT_RESET_COMPLETE event and check the reset result.
>>>>> + * If reset success, application should call rte_eth_dev_configure(
>>>>> + ),
>>>>> + * rte_eth_rx_queue_setup( ), rte_eth_tx_queue_setup( ),
>>>>> + * and rte_eth_dev_start( ) to reconfigure the device as appropriate.
>>>>> + *
>>>>> + * @Note
>>>>> + * To avoid unexpected behavior, the application should stop
>>>>> + calling
>>>>> + * Tx and Rx functions before calling rte_eth_dev_reset_async( ).
>>>>>   *
>>>>>   * @param port_id
>>>>>   *   The port identifier of the Ethernet device.
>>>>> @@ -1837,12 +1847,10 @@ void rte_eth_dev_close(uint16_t port_id);
>>>>>   *   - (0) if successful.
>>>>>   *   - (-EINVAL) if port identifier is invalid.
>>>>>   *   - (-ENOTSUP) if hardware doesn't support this function.
>>>>> - *   - (-EPERM) if not ran from the primary process.
>>>>> - *   - (-EIO) if re-initialisation failed or device is removed.
>>>>>   *   - (-ENOMEM) if the reset failed due to OOM.
>>>>> - *   - (-EAGAIN) if the reset temporarily failed and should be retried later.
>>>>> + *   - (<0) other errors from low level driver.
>>>>>   */
>>>>> -int rte_eth_dev_reset(uint16_t port_id);
>>>>> +int rte_eth_dev_reset_async(uint16_t port_id);
>>>>>
>>>>>  /**
>>>>>   * Enable receipt in promiscuous mode for an Ethernet device.
>>>>> @@ -2574,6 +2582,8 @@ enum rte_eth_event_type {
>>>>>  				/**< queue state event (enabled/disabled) */
>>>>>  	RTE_ETH_EVENT_INTR_RESET,
>>>>>  			/**< reset interrupt event, sent to VF on PF reset */
>>>>> +	RTE_ETH_EVENT_RESET_COMPLETE,
>>>>> +			/**< inform application that reset is completed */
>>>>>  	RTE_ETH_EVENT_VF_MBOX,  /**< message from the VF received by
>> PF
>>>> */
>>>>>  	RTE_ETH_EVENT_MACSEC,   /**< MACsec offload related event */
>>>>>  	RTE_ETH_EVENT_INTR_RMV, /**< device removal event */
>>>>>
>>>
>